Add Engine::disable_doc_comments and smarter doc-comments treatment.

RELEASES.md

@@ -33,12 +33,13 @@ New features
 ------------
 
 * `AST::iter_functions` now returns `ScriptFnMetadata` which includes, among others, _doc-comments_ for functions prefixed by `///` or `/**`.
-* A functions lookup cache is added to make function call resolution faster.
+* _Doc-comments_ can be enabled/disabled with the new `Engine::set_doc_comments` method.
 * A new feature `metadata` is added that pulls in `serde_json` and enables `Engine::gen_fn_metadata_to_json` which exports the full list of functions metadata (including those inside an `AST`) in JSON format.
 
 Enhancements
 ------------
 
+* A functions lookup cache is added to make function call resolution faster.
 * Capturing a constant variable in a closure is now supported, with no cloning.
 * Provides position info for `debug` statements.
 * A _look-ahead_ symbol is provided to custom syntax parsers, which can be used to parse variable-length symbol streams.

doc/src/SUMMARY.md

@@ -55,6 +55,7 @@ The Rhai Scripting Language
        2. [Export a Rust Function](plugins/function.md)
 5. [Rhai Language Reference](language/index.md)
    1. [Comments](language/comments.md)
+      1. [Doc-Comments](language/doc-comments.md)
    2. [Values and Types](language/values-and-types.md)
       1. [Dynamic Values](language/dynamic.md)
       2. [Serialization/Deserialization with `serde`](rust/serde.md)
@@ -79,14 +80,14 @@ The Rhai Scripting Language
    9. [If Statement](language/if.md)
    10. [Switch Expression](language/switch.md)
    11. [While Loop](language/while.md)
-   11. [Do Loop](language/do.md)
+   12. [Do Loop](language/do.md)
-   12. [Loop Statement](language/loop.md)
+   13. [Loop Statement](language/loop.md)
-   13. [For Loop](language/for.md)
+   14. [For Loop](language/for.md)
        1. [Iterators for Custom Types](language/iterator.md)
-   14. [Return Values](language/return.md)
+   15. [Return Values](language/return.md)
-   15. [Throw Exception on Error](language/throw.md)
+   16. [Throw Exception on Error](language/throw.md)
-   16. [Catch Exceptions](language/try-catch.md)
+   17. [Catch Exceptions](language/try-catch.md)
-   17. [Functions](language/functions.md)
+   18. [Functions](language/functions.md)
        1. [Call Method as Function](language/method.md)
        2. [Overloading](language/overload.md)
        3. [Namespaces](language/fn-namespaces.md)
@@ -94,11 +95,11 @@ The Rhai Scripting Language
        5. [Currying](language/fn-curry.md)
        6. [Anonymous Functions](language/fn-anon.md)
        7. [Closures](language/fn-closure.md)
-   18. [Print and Debug](language/print-debug.md)
+   19. [Print and Debug](language/print-debug.md)
-   19. [Modules](language/modules/index.md)
+   20. [Modules](language/modules/index.md)
        1. [Export Variables, Functions and Sub-Modules](language/modules/export.md)
        2. [Import Modules](language/modules/import.md)
-   20. [Eval Function](language/eval.md)
+   21. [Eval Function](language/eval.md)
 6. [Safety and Protection](safety/index.md)
    1. [Checked Arithmetic](safety/checked.md)
    2. [Sand-Boxing](safety/sandbox.md)

doc/src/engine/metadata/gen_fn_sig.md

@@ -48,7 +48,7 @@ In this case, the first parameter should be `&mut T` of the custom type and the
 
 > `set$prop(_, _, _)`
 
-### Script-defined functions
+### Script-Defined Functions
 
 Script-defined [function] signatures contain parameter names. Since all parameters, as well as
 the return value, are [`Dynamic`] the types are simply not shown.
@@ -69,7 +69,7 @@ is the same as:
 
 > `foo(x: Dynamic, y: Dynamic, z: Dynamic) -> Result<Dynamic, Box<EvalAltResult>>`
 
-### Plugin functions
+### Plugin Functions
 
 Functions defined in [plugin modules] are the best.  They contain all the metadata
 describing the functions.

doc/src/engine/options.md

@@ -5,14 +5,15 @@ Engine Configuration Options
 
 A number of other configuration options are available from the `Engine` to fine-tune behavior and safeguards.
 
-| Method                   | Not available under          | Description                                                                                                               |
+| Method                   | Not available under          | Description                                                                                                            |
-| ------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| ------------------------ | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
-| `set_optimization_level` | [`no_optimize`]              | sets the amount of script _optimizations_ performed. See [script optimization].                                           |
+| `set_doc_comments`       |                              | enables/disables [doc-comments]                                                                                        |
-| `set_max_expr_depths`    | [`unchecked`]                | sets the maximum nesting levels of an expression/statement. See [maximum statement depth].                                |
+| `set_optimization_level` | [`no_optimize`]              | sets the amount of script _optimizations_ performedSee [script optimization]                                           |
-| `set_max_call_levels`    | [`unchecked`]                | sets the maximum number of function call levels (default 50) to avoid infinite recursion. See [maximum call stack depth]. |
+| `set_max_expr_depths`    | [`unchecked`]                | sets the maximum nesting levels of an expression/statementSee [maximum statement depth]                                |
-| `set_max_operations`     | [`unchecked`]                | sets the maximum number of _operations_ that a script is allowed to consume. See [maximum number of operations].          |
+| `set_max_call_levels`    | [`unchecked`]                | sets the maximum number of function call levels (default 50) to avoid infinite recursionSee [maximum call stack depth] |
-| `set_max_modules`        | [`unchecked`]                | sets the maximum number of [modules] that a script is allowed to load. See [maximum number of modules].                   |
+| `set_max_operations`     | [`unchecked`]                | sets the maximum number of _operations_ that a script is allowed to consumeSee [maximum number of operations]          |
-| `set_max_string_size`    | [`unchecked`]                | sets the maximum length (in UTF-8 bytes) for [strings]. See [maximum length of strings].                                  |
+| `set_max_modules`        | [`unchecked`]                | sets the maximum number of [modules] that a script is allowed to loadSee [maximum number of modules]                   |
-| `set_max_array_size`     | [`unchecked`], [`no_index`]  | sets the maximum size for [arrays]. See [maximum size of arrays].                                                         |
+| `set_max_string_size`    | [`unchecked`]                | sets the maximum length (in UTF-8 bytes) for [strings]See [maximum length of strings]                                  |
-| `set_max_map_size`       | [`unchecked`], [`no_object`] | sets the maximum number of properties for [object maps]. See [maximum size of object maps].                               |
+| `set_max_array_size`     | [`unchecked`], [`no_index`]  | sets the maximum size for [arrays]See [maximum size of arrays]                                                         |
-| `disable_symbol`         |                              | disables a certain keyword or operator. See [disable keywords and operators].                                             |
+| `set_max_map_size`       | [`unchecked`], [`no_object`] | sets the maximum number of properties for [object maps]See [maximum size of object maps]                               |
+| `disable_symbol`         |                              | disables a certain keyword or operatorSee [disable keywords and operators]                                             |

doc/src/language/comments.md

@@ -22,43 +22,3 @@ let /* intruder comment */ name = "Bob";
    /*/*/*/*/**/*/*/*/*/
 */
 ```
-
-
-Doc-Comments
-------------
-
-Similar to Rust, comments starting with `///` (three slashes) or `/**` (two asterisks) are
-_doc-comments_.
-
-Doc-comments can only appear in front of [function] definitions, not any other elements:
-
-```rust
-/// This is a valid one-line doc-comment
-fn foo() {}
-
-/** This is a
- ** valid block
- ** doc-comment
- **/
-fn bar(x) {
-   /// Syntax error - this doc-comment is invalid
-   x + 1
-}
-
-/** Syntax error - this doc-comment is invalid */
-let x = 42;
-
-/// Syntax error - this doc-comment is also invalid
-{
-   let x = 42;
-}
-```
-
-Doc-comments are stored within the script's [`AST`] after compilation.
-
-The `AST::iter_functions` method provides a `ScriptFnMetadata` instance
-for each function defined within the script, which includes doc-comments.
-
-Doc-comments never affect the evaluation of a script nor do they incur
-significant performance overhead.  However, third party tools can take advantage
-of this information to auto-generate documentation for Rhai script functions.

doc/src/language/doc-comments.md

@@ -0,0 +1,76 @@
+Doc-Comments
+============
+
+Similar to Rust, comments starting with `///` (three slashes) or `/**` (two asterisks) are
+_doc-comments_.
+
+Doc-comments can only appear in front of [function] definitions, not any other elements:
+
+```rust
+/// This is a valid one-line doc-comment
+fn foo() {}
+
+/** This is a
+ ** valid block
+ ** doc-comment
+ **/
+fn bar(x) {
+   /// Syntax error - this doc-comment is invalid
+   x + 1
+}
+
+/** Syntax error - this doc-comment is invalid */
+let x = 42;
+
+/// Syntax error - this doc-comment is also invalid
+{
+   let x = 42;
+}
+```
+
+
+Special Cases
+-------------
+
+Long streams of `//////...` and `/*****...`  do _NOT_ form doc-comments.
+This is consistent with popular comment block styles for C-like languages.
+
+```rust
+///////////////////////////////  <- this is not a doc-comment
+// This is not a doc-comment //  <- this is a normal comment
+///////////////////////////////  <- this is not a doc-comment
+
+// However, watch out for comment lines starting with '///'
+
+//////////////////////////////////////////  <- this is not a doc-comment
+/// This, however, IS a doc-comment!!! ///  <- this starts with '///'
+//////////////////////////////////////////  <- this is not a doc-comment
+
+/****************************************
+ *                                      *
+ * This is also not a doc-comment block *
+ * so we don't have to put this in      *
+ * front of a function.                 *
+ *                                      *
+ ****************************************/
+```
+
+
+
+Using Doc-Comments
+------------------
+
+Doc-comments are stored within the script's [`AST`] after compilation.
+
+The `AST::iter_functions` method provides a `ScriptFnMetadata` instance
+for each function defined within the script, which includes doc-comments.
+
+Doc-comments never affect the evaluation of a script nor do they incur
+significant performance overhead.  However, third party tools can take advantage
+of this information to auto-generate documentation for Rhai script functions.
+
+
+Disabling Doc-Comments
+----------------------
+
+Doc-comments can be disabled via the `Engine::set_doc_comments` method.

doc/src/links.md

@@ -91,7 +91,7 @@
 [timestamp]: {{rootUrl}}/language/timestamps.md
 [timestamps]: {{rootUrl}}/language/timestamps.md
 
-[doc-comments]: {{rootUrl}}/language/comments.md#doc-comments
+[doc-comments]: {{rootUrl}}/language/doc-comments.md
 [function]: {{rootUrl}}/language/functions.md
 [functions]: {{rootUrl}}/language/functions.md
 [function overloading]: {{rootUrl}}/rust/functions.md#function-overloading

examples/repl.rs

@@ -147,6 +147,15 @@ fn main() {
                 println!();
                 continue;
             }
+            // "json" => {
+            //     println!(
+            //         "{}",
+            //         engine
+            //             .gen_fn_metadata_to_json(Some(&main_ast), false)
+            //             .unwrap()
+            //     );
+            //     continue;
+            // }
             _ => (),
         }
 

src/engine.rs

@@ -654,6 +654,9 @@ pub struct Engine {
     /// Max limits.
     #[cfg(not(feature = "unchecked"))]
     pub(crate) limits: Limits,
+
+    /// Disable doc-comments?
+    pub(crate) disable_doc_comments: bool,
 }
 
 impl fmt::Debug for Engine {
@@ -794,6 +797,8 @@ impl Engine {
                 #[cfg(not(feature = "no_object"))]
                 max_map_size: 0,
             },
+
+            disable_doc_comments: false,
         };
 
         engine.load_package(StandardPackage::new().get());
@@ -847,6 +852,8 @@ impl Engine {
                 #[cfg(not(feature = "no_object"))]
                 max_map_size: 0,
             },
+
+            disable_doc_comments: false,
         }
     }
 

src/engine_settings.rs

@@ -40,6 +40,12 @@ impl Engine {
     pub fn optimization_level(&self) -> crate::OptimizationLevel {
         self.optimization_level
     }
+    /// Enable/disable doc-comments.
+    #[inline(always)]
+    pub fn set_doc_comments(&mut self, enable: bool) -> &mut Self {
+        self.disable_doc_comments = !enable;
+        self
+    }
     /// Set the maximum levels of function calls allowed for a script in order to avoid
     /// infinite recursion and stack overflows.
     #[cfg(not(feature = "unchecked"))]

src/token.rs

@@ -354,9 +354,7 @@ impl Token {
             Reserved(s) => s.clone().into(),
             Custom(s) => s.clone().into(),
             LexError(err) => err.to_string().into(),
-
+            Comment(s) => s.clone().into(),
-            Comment(s) if is_doc_comment(s) => s[..3].to_string().into(),
-            Comment(s) => s[..2].to_string().into(),
 
             token => match token {
                 LeftBrace => "{",
@@ -759,6 +757,8 @@ pub struct TokenizeState {
     pub end_with_none: bool,
     /// Include comments?
     pub include_comments: bool,
+    /// Disable doc-comments?
+    pub disable_doc_comments: bool,
 }
 
 /// _(INTERNALS)_ Trait that encapsulates a peekable character input stream.
@@ -1020,7 +1020,8 @@ fn is_binary_char(c: char) -> bool {
 /// Test if the comment block is a doc-comment.
 #[inline(always)]
 pub fn is_doc_comment(comment: &str) -> bool {
-    comment.starts_with("///") || comment.starts_with("/**")
+    (comment.starts_with("///") && !comment.starts_with("////"))
+        || (comment.starts_with("/**") && !comment.starts_with("/***"))
 }
 
 /// Get the next token.
@@ -1040,7 +1041,9 @@ fn get_next_token_inner(
 
         state.comment_level = scan_block_comment(stream, state.comment_level, pos, &mut comment);
 
-        if state.include_comments || is_doc_comment(comment.as_ref().unwrap()) {
+        if state.include_comments
+            || (!state.disable_doc_comments && is_doc_comment(comment.as_ref().unwrap()))
+        {
             return Some((Token::Comment(comment.unwrap()), start_pos));
         }
     }
@@ -1288,9 +1291,14 @@ fn get_next_token_inner(
                 eat_next(stream, pos);
 
                 let mut comment = match stream.peek_next() {
-                    Some('/') => {
+                    Some('/') if !state.disable_doc_comments => {
                         eat_next(stream, pos);
-                        Some("///".to_string())
+
+                        // Long streams of `///...` are not doc-comments
+                        match stream.peek_next() {
+                            Some('/') => None,
+                            _ => Some("///".to_string()),
+                        }
                     }
                     _ if state.include_comments => Some("//".to_string()),
                     _ => None,
@@ -1316,9 +1324,14 @@ fn get_next_token_inner(
                 eat_next(stream, pos);
 
                 let mut comment = match stream.peek_next() {
-                    Some('*') => {
+                    Some('*') if !state.disable_doc_comments => {
                         eat_next(stream, pos);
-                        Some("/**".to_string())
+
+                        // Long streams of `/****...` are not doc-comments
+                        match stream.peek_next() {
+                            Some('*') => None,
+                            _ => Some("/**".to_string()),
+                        }
                     }
                     _ if state.include_comments => Some("/*".to_string()),
                     _ => None,
@@ -1785,6 +1798,7 @@ impl Engine {
                 comment_level: 0,
                 end_with_none: false,
                 include_comments: false,
+                disable_doc_comments: self.disable_doc_comments,
             },
             pos: Position::new(1, 0),
             stream: MultiInputsStream {

tests/comments.rs

@@ -29,7 +29,7 @@ fn test_comments() -> Result<(), Box<EvalAltResult>> {
 #[cfg(not(feature = "no_function"))]
 #[test]
 fn test_comments_doc() -> Result<(), Box<EvalAltResult>> {
-    let engine = Engine::new();
+    let mut engine = Engine::new();
 
     let ast = engine.compile(
         r"
@@ -54,6 +54,16 @@ fn test_comments_doc() -> Result<(), Box<EvalAltResult>> {
         )
         .is_err());
 
+    engine.compile(
+        r"
+            ///////////////
+            let x = 42;
+
+            /***************/
+            let x = 42;
+        ",
+    )?;
+
     let ast = engine.compile(
         r"
             /** Hello world
@@ -78,5 +88,17 @@ fn test_comments_doc() -> Result<(), Box<EvalAltResult>> {
         )
         .is_err());
 
+    engine.set_doc_comments(false);
+
+    engine.compile(
+        r"
+            /// Hello world!
+            let x = 42;
+
+            /** Hello world! */
+            let x = 42;
+        ",
+    )?;
+
     Ok(())
 }