Enable custom syntax without internals.

RELEASES.md

@@ -15,6 +15,7 @@ New features
 * Disallow many keywords as variables, such as `print`, `eval`, `call`, `this` etc.
 * `x.call(f, ...)` allows binding `x` to `this` for the function referenced by the function pointer `f`.
 * Anonymous functions in the syntax of a closure, e.g. `|x, y, z| x + y - z`.
+* Custom syntax now works even without the `internals` feature.
 
 Breaking changes
 ----------------
@@ -48,7 +49,7 @@ Breaking changes
 New features
 ------------
 
-* New `serde` feature to allow serializating/deserializating to/from `Dynamic` values using [`serde`](https://crates.io/crates/serde).
+* New `serde` feature to allow serializing/deserializing to/from `Dynamic` values using [`serde`](https://crates.io/crates/serde).
   This is particularly useful when converting a Rust `struct` to a `Dynamic` _object map_ and back.
 * `Engine::disable_symbol` to surgically disable keywords and/or operators.
 * `Engine::register_custom_operator` to define a custom operator.

doc/src/engine/custom-syntax.md

@@ -33,18 +33,7 @@ Where This Might Be Useful
 * Where you just want to confuse your user and make their lives miserable, because you can.
 
 
-Step One - Start With `internals`
+Step One - Design The Syntax
---------------------------------
-
-Since a custom syntax taps deeply into the `AST` and evaluation process of the `Engine`,
-the [`internals`] feature must be on in order to expose these necessary internal data structures.
-
-Beware that Rhai internal data structures are _volatile_ and may change without warning.
-
-Caveat emptor.
-
-
-Step Two - Design The Syntax
 ---------------------------
 
 A custom syntax is simply a list of symbols.
@@ -116,8 +105,8 @@ print(hello);   // variable declared by a custom syntax persists!
 ```
 
 
-Step Three - Implementation
+Step Two - Implementation
---------------------------
+-------------------------
 
 Any custom syntax must include an _implementation_ of it.
 
@@ -164,8 +153,6 @@ let expr = inputs.get(0).unwrap();
 let result = engine.eval_expression_tree(context, scope, expr)?;
 ```
 
-As can be seem above, most arguments are simply passed straight-through to `engine::eval_expression_tree`.
-
 ### Declare Variables
 
 New variables maybe declared (usually with a variable name that is passed in via `$ident$).
@@ -179,14 +166,14 @@ In other words, any `scope.push(...)` calls must come _before_ any `engine::eval
 let var_name = inputs[0].get_variable_name().unwrap().to_string();
 let expr = inputs.get(1).unwrap();
 
-scope.push(var_name, 0 as INT);     // do this BEFORE engine.eval_expression_tree!
+scope.push(var_name, 0 as INT);     // do this BEFORE 'engine.eval_expression_tree'!
 
-let result = engine.eval_expression_tree(scope, mods, state, lib, this_ptr, expr, level)?;
+let result = engine.eval_expression_tree(context, scope, expr)?;
 ```
 
 
-Step Four - Register the Custom Syntax
+Step Three - Register the Custom Syntax
--------------------------------------
+--------------------------------------
 
 Use `Engine::register_custom_syntax` to register a custom syntax.
 
@@ -236,7 +223,7 @@ engine.register_custom_syntax(
 ```
 
 
-Step Five - Disable Unneeded Statement Types
+Step Four - Disable Unneeded Statement Types
 -------------------------------------------
 
 When a DSL needs a custom syntax, most likely than not it is extremely specialized.
@@ -254,13 +241,13 @@ custom syntax (plus possibly expressions).  But again, Don't Do It™ - unless y
 of what you're doing.
 
 
-Step Six - Document
+Step Five - Document
--------------------
+--------------------
 
 For custom syntax, documentation is crucial.
 
 Make sure there are _lots_ of examples for users to follow.
 
 
-Step Seven - Profit!
+Step Six - Profit!
---------------------
+------------------

doc/src/engine/dsl.md

@@ -11,7 +11,7 @@ Expressions Only
 
 In many DSL scenarios, only evaluation of expressions is needed.
 
-The `Engine::eval_expression_XXX`[`eval_expression`] API can be used to restrict
+The [`Engine::eval_expression_XXX`][`eval_expression`] API can be used to restrict
 a script to expressions only.
 
 
@@ -21,8 +21,7 @@ Disable Keywords and/or Operators
 In some DSL scenarios, it is necessary to further restrict the language to exclude certain
 language features that are not necessary or dangerous to the application.
 
-For example, a DSL may disable the `while` loop altogether while keeping all other statement
+For example, a DSL may disable the `while` loop while keeping all other statement types intact.
-types intact.
 
 It is possible, in Rhai, to surgically [disable keywords and operators].
 
@@ -54,31 +53,29 @@ Custom Syntax
 For advanced DSL scenarios, it is possible to define entire expression [_syntax_][custom syntax] -
 essentially custom statement types.
 
-The [`internals`] feature is needed to be able to define [custom syntax] in Rhai.
+For example, the following is a SQL-like syntax for some obscure DSL operation:
-
-For example, the following is a SQL like syntax for some obscure DSL operation:
 
 ```rust
 let table = [..., ..., ..., ...];
 
-// Syntax = "calculate" $ident$ $ident$ "from" $expr$ "->" $ident$ ":" $expr$
+// Syntax = calculate $ident$ $ident$ from $expr$ -> $ident$ : $expr$
 let total = calculate sum price from table -> row : row.weight > 50;
 
-// Note: There is nothing special about the use of symbols; to make it look exactly like SQL:
+// Note: There is nothing special about those symbols; to make it look exactly like SQL:
-// Syntax = "SELECT" $ident$ "(" $ident$ ")" "FROM" $expr$ "AS" $ident$ "WHERE" $expr$
+// Syntax = SELECT $ident$ ( $ident$ ) FROM $expr$ AS $ident$ WHERE $expr$
 let total = SELECT sum(price) FROM table AS row WHERE row.weight > 50;
 ```
 
 After registering this custom syntax with Rhai, it can be used anywhere inside a script as
 a normal expression.
 
-For its evaluation, the callback function will receive the following list of parameters:
+For its evaluation, the callback function will receive the following list of inputs:
 
-`exprs[0] = "sum"` - math operator  
+* `inputs[0] = "sum"` - math operator
-`exprs[1] = "price"` - field name  
+* `inputs[1] = "price"` - field name
-`exprs[2] = Expression(table)` - data source  
+* `inputs[2] = Expression(table)` - data source
-`exprs[3] = "row"` - loop variable name  
+* `inputs[3] = "row"` - loop variable name
-`exprs[4] = Expression(row.wright > 50)` - expression  
+* `inputs[4] = Expression(row.wright > 50)` - filter predicate
 
-The other identified, such as `"select"`, `"from"`, as as as symbols `->` and `:` are
+Other identifiers, such as `"calculate"`, `"from"`, as well as symbols such as `->` and `:`,
-parsed in the order defined within the custom syntax.
+are parsed in the order defined within the custom syntax.

src/engine.rs

@@ -11,15 +11,13 @@ use crate::parser::{Expr, FnAccess, ImmutableString, ReturnType, ScriptFnDef, St
 use crate::r#unsafe::unsafe_cast_var_name_to_lifetime;
 use crate::result::EvalAltResult;
 use crate::scope::{EntryType as ScopeEntryType, Scope};
+use crate::syntax::{CustomSyntax, EvalContext, Expression};
 use crate::token::Position;
 use crate::utils::StaticVec;
 
 #[cfg(not(feature = "no_float"))]
 use crate::parser::FLOAT;
 
-#[cfg(feature = "internals")]
-use crate::syntax::{CustomSyntax, EvalContext};
-
 use crate::stdlib::{
     any::{type_name, TypeId},
     borrow::Cow,
@@ -88,45 +86,10 @@ pub const FN_SET: &str = "set$";
 pub const FN_IDX_GET: &str = "index$get$";
 pub const FN_IDX_SET: &str = "index$set$";
 pub const FN_ANONYMOUS: &str = "anon$";
-
-#[cfg(feature = "internals")]
 pub const MARKER_EXPR: &str = "$expr$";
-#[cfg(feature = "internals")]
 pub const MARKER_BLOCK: &str = "$block$";
-#[cfg(feature = "internals")]
 pub const MARKER_IDENT: &str = "$ident$";
 
-#[cfg(feature = "internals")]
-#[derive(Debug, Clone, Hash)]
-pub struct Expression<'a>(&'a Expr);
-
-#[cfg(feature = "internals")]
-impl<'a> From<&'a Expr> for Expression<'a> {
-    fn from(expr: &'a Expr) -> Self {
-        Self(expr)
-    }
-}
-
-#[cfg(feature = "internals")]
-impl Expression<'_> {
-    /// If this expression is a variable name, return it.  Otherwise `None`.
-    #[cfg(feature = "internals")]
-    pub fn get_variable_name(&self) -> Option<&str> {
-        match self.0 {
-            Expr::Variable(x) => Some((x.0).0.as_str()),
-            _ => None,
-        }
-    }
-    /// Get the expression.
-    pub(crate) fn expr(&self) -> &Expr {
-        &self.0
-    }
-    /// Get the position of this expression.
-    pub fn position(&self) -> Position {
-        self.0.position()
-    }
-}
-
 /// A type specifying the method of chaining.
 #[derive(Debug, Clone, Copy, Eq, PartialEq, Hash)]
 enum ChainType {
@@ -316,7 +279,6 @@ pub struct Engine {
     /// A hashset containing custom keywords and precedence to recognize.
     pub(crate) custom_keywords: Option<HashMap<String, u8>>,
     /// Custom syntax.
-    #[cfg(feature = "internals")]
     pub(crate) custom_syntax: Option<HashMap<String, CustomSyntax>>,
 
     /// Callback closure for implementing the `print` command.
@@ -376,8 +338,6 @@ impl Default for Engine {
             type_names: None,
             disabled_symbols: None,
             custom_keywords: None,
-
-            #[cfg(feature = "internals")]
             custom_syntax: None,
 
             // default print/debug implementations
@@ -605,8 +565,6 @@ impl Engine {
             type_names: None,
             disabled_symbols: None,
             custom_keywords: None,
-
-            #[cfg(feature = "internals")]
             custom_syntax: None,
 
             print: Box::new(|_| {}),
@@ -1734,8 +1692,6 @@ impl Engine {
     /// ## WARNING - Low Level API
     ///
     /// This function is very low level.  It evaluates an expression from an AST.
-    #[cfg(feature = "internals")]
-    #[deprecated(note = "this method is volatile and may change")]
     pub fn eval_expression_tree(
         &self,
         context: &mut EvalContext,
@@ -2215,7 +2171,6 @@ impl Engine {
             Expr::False(_) => Ok(false.into()),
             Expr::Unit(_) => Ok(().into()),
 
-            #[cfg(feature = "internals")]
             Expr::Custom(x) => {
                 let func = (x.0).1.as_ref();
                 let ep: StaticVec<_> = (x.0).0.iter().map(|e| e.into()).collect();

src/lib.rs

@@ -91,7 +91,6 @@ mod scope;
 mod serde;
 mod settings;
 mod stdlib;
-#[cfg(feature = "internals")]
 mod syntax;
 mod token;
 mod r#unsafe;
@@ -106,6 +105,7 @@ pub use module::Module;
 pub use parser::{ImmutableString, AST, INT};
 pub use result::EvalAltResult;
 pub use scope::Scope;
+pub use syntax::{EvalContext, Expression};
 pub use token::Position;
 pub use utils::calc_fn_spec as calc_fn_hash;
 
@@ -169,11 +169,7 @@ pub use parser::{CustomExpr, Expr, ReturnType, ScriptFnDef, Stmt};
 
 #[cfg(feature = "internals")]
 #[deprecated(note = "this type is volatile and may change")]
-pub use engine::{Expression, Imports, State as EvalState};
+pub use engine::{Imports, State as EvalState};
-
-#[cfg(feature = "internals")]
-#[deprecated(note = "this type is volatile and may change")]
-pub use syntax::EvalContext;
 
 #[cfg(feature = "internals")]
 #[deprecated(note = "this type is volatile and may change")]

src/parser.rs

@@ -2,23 +2,19 @@
 
 use crate::any::{Dynamic, Union};
 use crate::calc_fn_hash;
-use crate::engine::{make_getter, make_setter, Engine, FN_ANONYMOUS, KEYWORD_THIS};
+use crate::engine::{
+    make_getter, make_setter, Engine, FN_ANONYMOUS, KEYWORD_THIS, MARKER_BLOCK, MARKER_EXPR,
+    MARKER_IDENT,
+};
 use crate::error::{LexError, ParseError, ParseErrorType};
+use crate::fn_native::Shared;
 use crate::module::{Module, ModuleRef};
 use crate::optimize::{optimize_into_ast, OptimizationLevel};
 use crate::scope::{EntryType as ScopeEntryType, Scope};
+use crate::syntax::FnCustomSyntaxEval;
 use crate::token::{is_valid_identifier, Position, Token, TokenStream};
 use crate::utils::{StaticVec, StraightHasherBuilder};
 
-#[cfg(feature = "internals")]
-use crate::engine::{MARKER_BLOCK, MARKER_EXPR, MARKER_IDENT};
-
-#[cfg(feature = "internals")]
-use crate::fn_native::Shared;
-
-#[cfg(feature = "internals")]
-use crate::syntax::FnCustomSyntaxEval;
-
 use crate::stdlib::{
     borrow::Cow,
     boxed::Box,
@@ -587,17 +583,14 @@ impl Stmt {
 }
 
 #[derive(Clone)]
-#[cfg(feature = "internals")]
 pub struct CustomExpr(pub StaticVec<Expr>, pub Shared<FnCustomSyntaxEval>);
 
-#[cfg(feature = "internals")]
 impl fmt::Debug for CustomExpr {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         fmt::Debug::fmt(&self.0, f)
     }
 }
 
-#[cfg(feature = "internals")]
 impl Hash for CustomExpr {
     fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
         self.0.hash(state);
@@ -683,7 +676,6 @@ pub enum Expr {
     /// ()
     Unit(Position),
     /// Custom syntax
-    #[cfg(feature = "internals")]
     Custom(Box<(CustomExpr, Position)>),
 }
 
@@ -781,7 +773,6 @@ impl Expr {
 
             Self::Dot(x) | Self::Index(x) => x.0.position(),
 
-            #[cfg(feature = "internals")]
             Self::Custom(x) => x.1,
         }
     }
@@ -816,8 +807,6 @@ impl Expr {
             Self::Assignment(x) => x.3 = new_pos,
             Self::Dot(x) => x.2 = new_pos,
             Self::Index(x) => x.2 = new_pos,
-
-            #[cfg(feature = "internals")]
             Self::Custom(x) => x.1 = new_pos,
         }
 
@@ -925,7 +914,6 @@ impl Expr {
                 _ => false,
             },
 
-            #[cfg(feature = "internals")]
             Self::Custom(_) => false,
         }
     }
@@ -2148,7 +2136,6 @@ fn parse_expr(
     settings.ensure_level_within_max_limit(state.max_expr_depth)?;
 
     // Check if it is a custom syntax.
-    #[cfg(feature = "internals")]
     if let Some(ref custom) = state.engine.custom_syntax {
         let (token, pos) = input.peek().unwrap();
         let token_pos = *pos;

src/syntax.rs

@@ -1,14 +1,13 @@
 //! Module containing implementation for custom syntax.
-#![cfg(feature = "internals")]
-
 use crate::any::Dynamic;
-use crate::engine::{Engine, Expression, Imports, State, MARKER_BLOCK, MARKER_EXPR, MARKER_IDENT};
+use crate::engine::{Engine, Imports, State, MARKER_BLOCK, MARKER_EXPR, MARKER_IDENT};
-use crate::error::LexError;
+use crate::error::{LexError, ParseError, ParseErrorType};
 use crate::fn_native::{SendSync, Shared};
 use crate::module::Module;
+use crate::parser::Expr;
 use crate::result::EvalAltResult;
 use crate::scope::Scope;
-use crate::token::{is_valid_identifier, Token};
+use crate::token::{is_valid_identifier, Position, Token};
 use crate::utils::StaticVec;
 
 use crate::stdlib::{
@@ -32,6 +31,33 @@ pub type FnCustomSyntaxEval = dyn Fn(&Engine, &mut EvalContext, &mut Scope, &[Ex
     + Send
     + Sync;
 
+#[derive(Debug, Clone, Hash)]
+pub struct Expression<'a>(&'a Expr);
+
+impl<'a> From<&'a Expr> for Expression<'a> {
+    fn from(expr: &'a Expr) -> Self {
+        Self(expr)
+    }
+}
+
+impl Expression<'_> {
+    /// If this expression is a variable name, return it.  Otherwise `None`.
+    pub fn get_variable_name(&self) -> Option<&str> {
+        match self.0 {
+            Expr::Variable(x) => Some((x.0).0.as_str()),
+            _ => None,
+        }
+    }
+    /// Get the expression.
+    pub(crate) fn expr(&self) -> &Expr {
+        &self.0
+    }
+    /// Get the position of this expression.
+    pub fn position(&self) -> Position {
+        self.0.position()
+    }
+}
+
 #[derive(Clone)]
 pub struct CustomSyntax {
     pub segments: StaticVec<String>,
@@ -45,6 +71,7 @@ impl fmt::Debug for CustomSyntax {
     }
 }
 
+#[derive(Debug)]
 pub struct EvalContext<'a, 'b: 'a, 's, 'm, 't, 'd: 't> {
     pub(crate) mods: &'a mut Imports<'b>,
     pub(crate) state: &'s mut State,
@@ -56,7 +83,7 @@ pub struct EvalContext<'a, 'b: 'a, 's, 'm, 't, 'd: 't> {
 impl Engine {
     pub fn register_custom_syntax<S: AsRef<str> + ToString>(
         &mut self,
-        value: &[S],
+        keywords: &[S],
         scope_delta: isize,
         func: impl Fn(
                 &Engine,
@@ -66,15 +93,18 @@ impl Engine {
             ) -> Result<Dynamic, Box<EvalAltResult>>
             + SendSync
             + 'static,
-    ) -> Result<&mut Self, Box<LexError>> {
+    ) -> Result<&mut Self, Box<ParseError>> {
-        if value.is_empty() {
-            return Err(Box::new(LexError::ImproperSymbol("".to_string())));
-        }
-
         let mut segments: StaticVec<_> = Default::default();
 
-        for s in value {
+        for s in keywords {
-            let seg = match s.as_ref() {
+            let s = s.as_ref().trim();
+
+            // skip empty keywords
+            if s.is_empty() {
+                continue;
+            }
+
+            let seg = match s {
                 // Markers not in first position
                 MARKER_EXPR | MARKER_BLOCK | MARKER_IDENT if !segments.is_empty() => s.to_string(),
                 // Standard symbols not in first position
@@ -115,12 +145,25 @@ impl Engine {
                     s.into()
                 }
                 // Anything else is an error
-                _ => return Err(Box::new(LexError::ImproperSymbol(s.to_string()))),
+                _ => {
+                    return Err(LexError::ImproperSymbol(format!(
+                        "Improper symbol for custom syntax: '{}'",
+                        s
+                    ))
+                    .into_err(Position::none())
+                    .into());
+                }
             };
 
             segments.push(seg);
         }
 
+        // If the syntax has no keywords, just ignore the registration
+        if segments.is_empty() {
+            return Ok(self);
+        }
+
+        // Remove the first keyword as the discriminator
         let key = segments.remove(0);
 
         let syntax = CustomSyntax {

tests/syntax.rs

@@ -1,5 +1,6 @@
-#![cfg(feature = "internals")]
+use rhai::{
-use rhai::{Engine, EvalAltResult, EvalContext, Expression, LexError, ParseErrorType, Scope, INT};
+    Engine, EvalAltResult, EvalContext, Expression, ParseError, ParseErrorType, Scope, INT,
+};
 
 #[test]
 fn test_custom_syntax() -> Result<(), Box<EvalAltResult>> {
@@ -72,7 +73,7 @@ fn test_custom_syntax() -> Result<(), Box<EvalAltResult>> {
     // The first symbol must be an identifier
     assert!(matches!(
         *engine.register_custom_syntax(&["!"], 0, |_, _, _, _| Ok(().into())).expect_err("should error"),
-        LexError::ImproperSymbol(s) if s == "!"
+        ParseError(err, _) if *err == ParseErrorType::BadInput("Improper symbol for custom syntax: '!'".to_string())
     ));
 
     Ok(())