Recovering segments from pygmetized code fails for some lexers - #143
Recovering segments from pygmetized code fails for some lexers#143shiftyp wants to merge 1 commit into
Conversation
…hat there will be a single space between SEGMENT and DIVIDER. This fails for some lexers. Relaxing this assumption fixes the issue.
There was a problem hiding this comment.
Using .? seems a bit dangerous. It could match Anything. \s would be safer.
There was a problem hiding this comment.
I agree its not ideal, but for some languages, JSP for instance, the words SEGMENT and DIVIDER are in separate spans. It might be possible to come up with a more exact regex. I'll try and post an example of the parsed JSP comment HTML today and we can discuss options.
There was a problem hiding this comment.
they wind up in separate spans??? wow... I wasn't expecting that...
There was a problem hiding this comment.
this is especially disconcerting because ORIGINALLY "#{seg}\s#{div}" was just a literal "SEGMENT DIVIDER"
I split it up into 2 words like this to prevent an error when running groc on itself.
There was a problem hiding this comment.
you need this for the xml/html language support?
I suspect that the lexer might be doing something wierd... hmm...
There was a problem hiding this comment.
one option might be: (?:\s|INSERT_VERY_SPECIFIC_REGEX_FOR_THIS_ISSUE) so we match either a single space Or the extra close/open span
There was a problem hiding this comment.
I suppose .*? was a bit lazy of me, sorry. Here is the output for JSP comments:
<span class="k"><%-</span><span class="o">-</span> <span class="n">SEGMENT</span> <span class="n">DIVIDER</span> <span class="o">--</span><span class="k">%></span>
There was a problem hiding this comment.
so </span> <span class="n"> is the bad stuff we need to match
(?:\s|<\/span>\s*<span\sclass="n">)
There was a problem hiding this comment.
Yeah, that would work. I'm not sure if the same pattern applies to comments in other languages though. I'll have to check XML and HTML.
|
Please read this comment: #134 (comment) If we implement the code-segmentation in the proper tool, which is to my mind the syntax-highlighter, we would not just simplify groc's implementation, because groc would get a well known JSON-formatted stream of raw-comments and formatted-code, but also raise groc's precision dramatically. Another benefit of this approach is, to add new syntax-highlighters, we just need to implement this kind of glue-API that spits out the preprocessed JSON-stream. Possible candidates: ruby's rouge or node's highlight.js. As rouge uses the same styles as pygments, only highlight.js would have a different styling. |
3 participants
The regex that splits the segments from the pygmetized code assumes that there will be a single space between SEGMENT and DIVIDER. This fails for some lexers. Relaxing this assumption fixes the issue.