HRINT-4790/4788/4789: SEO analysis, GenAI spam check, social media posting#138
HRINT-4790/4788/4789: SEO analysis, GenAI spam check, social media posting#138ayende wants to merge 35 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR replaces Akismet-based spam filtering with a RavenDB GenAI pipeline, adds AI-generated SEO metadata (meta description, keywords, JSON-LD structured data) to posts, introduces auto-publishing of posts to Twitter/Reddit driven by @social tag conventions and RavenDB subscriptions, and tweaks LESS so in-body headings sit below the post title. Comments are now created in a Pending state, evaluated by a spam-filter GenAI task, and either accepted (sending a NewComment email) or moved to spam (accumulating into a daily digest). Email delivery and social posting move from FluentScheduler/TaskExecutor to RavenDB data subscriptions (email-worker, social-posting).
Changes:
- Add GenAI tasks (
spam-filter,seo-analysis,social-media) registered at startup, plusEmailSubscriptionandSocialPostingSubscriptionworkers; remove Akismet service,AddCommentTask,SendEmailTask,BackgroundTask/TaskExecutor. - Extend
PostwithSeo/Socialsub-models,PostComments.CommentwithSpamCheckStatus,BlogConfigwithTwitterBearerToken; surface SEO data in admin and public views, and filter@-prefixed tags from public mappers. - Delete
Web.config/Web.Debug.config/Web.Release.config, drop theJoel.Net.Akismetreference, addScriban7.2.0, and add LESS rules demotingh1..h6inside.text-wrapper.
Reviewed changes
Copilot reviewed 41 out of 42 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| RaccoonBlog.Web/wwwroot/css/styles.less | Demote body heading sizes inside .text-wrapper. |
| RaccoonBlog.Web/Web.Release.config / Web.Debug.config / Web.config | Delete IIS/ASP.NET classic config files. |
| RaccoonBlog.Web/Views/Welcome/Index.cshtml | Remove Akismet key field from welcome form. |
| RaccoonBlog.Web/Views/PostDetails/Details.cshtml | Use AI meta description with fallback, render keywords + JSON-LD, hide @-prefixed related tags. |
| RaccoonBlog.Web/ViewModels/SpamDigestEmailViewModel.cs | New view model for daily spam digest email. |
| RaccoonBlog.Web/ViewModels/PostViewModel.cs | Add SEO fields to public post view model. |
| RaccoonBlog.Web/ViewModels/NewCommentEmailViewModel.cs | Add SpamCheckStatus field. |
| RaccoonBlog.Web/ViewModels/AdminPostDetailsViewModel.cs | Surface SpamCheckStatus on admin comment and SEO fields on post. |
| RaccoonBlog.Web/Services/AkismetService.cs | Delete Akismet service. |
| RaccoonBlog.Web/RaccoonBlog.Web.csproj | Drop Joel.Net.Akismet reference; add Scriban package. |
| RaccoonBlog.Web/Program.cs | Register GenAI tasks, enable document refresh, start subscriptions; drop AutoMapper/auth/index using/cert callback. |
| RaccoonBlog.Web/Models/SocialMedia.cs | New embedded model with Twitter/Reddit text and auto-publish flags. |
| RaccoonBlog.Web/Models/SeoMetadata.cs | New embedded SEO model. |
| RaccoonBlog.Web/Models/SendEmailCommand.cs | New EmailCommands document shape (incl. SpamDigest entries). |
| RaccoonBlog.Web/Models/PostComments.cs | Add SpamCheckStatus enum and field on Comment. |
| RaccoonBlog.Web/Models/Post.cs | Add Social, Seo, and SEO-related PostInput properties. |
| RaccoonBlog.Web/Models/BlogConfig.cs | Add TwitterBearerToken setting. |
| RaccoonBlog.Web/Infrastructure/Tasks/* | Delete TaskExecutor, BackgroundTask, SendEmailTask, AddCommentTask. |
| RaccoonBlog.Web/Infrastructure/SocialPostingSubscription.cs | New subscription worker that posts to Twitter (stub Reddit). |
| RaccoonBlog.Web/Infrastructure/Indexes/PostComments_CreationDate.cs | Exclude Pending comments from the index. |
| RaccoonBlog.Web/Infrastructure/GenAiTasks/SpamFilterGenAiTask.cs | Register spam-filter GenAI task + JS update script (creates emails/digest docs). |
| RaccoonBlog.Web/Infrastructure/GenAiTasks/SocialMediaGenAiTask.cs | Register social-media GenAI task. |
| RaccoonBlog.Web/Infrastructure/GenAiTasks/SeoAnalysisGenAiTask.cs | Register SEO-analysis GenAI task. |
| RaccoonBlog.Web/Infrastructure/EmailSubscription.cs | New subscription worker + Scriban templates for emails. |
| RaccoonBlog.Web/Infrastructure/AutoMapper/Profiles/Resolvers/TagsResolver.cs | Guard against null/empty tag collections. |
| RaccoonBlog.Web/Infrastructure/AutoMapper/Profiles/PostViewModelMapperProfile.cs | Map SEO fields and filter @ tags from public mapping. |
| RaccoonBlog.Web/Infrastructure/AutoMapper/Profiles/PostsViewModelMapperProfile.cs | Filter @ tags. |
| RaccoonBlog.Web/Infrastructure/AutoMapper/Profiles/PostsAdminViewModelMapperProfile.cs | Map SEO fields between PostInput/Post/Admin view models. |
| RaccoonBlog.Web/Infrastructure/AutoMapper/AutoMapperConfiguration.cs | Remove HttpRequest -> AddCommentTask.RequestValues map. |
| RaccoonBlog.Web/Helpers/SeoHelper.cs | New helper to emit JSON-LD BlogPosting. |
| RaccoonBlog.Web/Helpers/RedditHelper.cs | Include new social/social-reddit tags and respect Social.DisableAutoPublish. |
| RaccoonBlog.Web/Controllers/PostDetailsController.cs | Inline comment creation as Pending; hide pending comments from non-authors. |
| RaccoonBlog.Web/Areas/Admin/Views/Settings/Index.cshtml | Drop Akismet field; add Twitter bearer token field. |
| RaccoonBlog.Web/Areas/Admin/Views/Posts/Edit.cshtml | Read-only SEO panel. |
| RaccoonBlog.Web/Areas/Admin/Views/Posts/Details.cshtml | Show SEO summary and SpamCheckStatus badges. |
| RaccoonBlog.Web/Areas/Admin/Controllers/PostsController.cs | Replace Akismet calls with SpamCheckStatus updates in admin moderation. |
Comments suppressed due to low confidence (1)
RaccoonBlog.Web/Web.config:1
Web.config,Web.Debug.config, andWeb.Release.configare being deleted as part of this PR, but the PR description focuses on SEO/spam/social media work and contains no rationale for removing the IIS/ASP.NET configuration files. These files configure the Raven connection string for production, mail/SMTP settings, MIME types, assembly binding redirects,dotlesshandler, machine key, etc. Removing them outright will break Release deployments (noRaven/Urls, no MIME maps, no binding redirects). If the project has migrated entirely offSystem.Web/IIS classic config toappsettings.jsonthis should be called out explicitly in the PR description; otherwise these deletions look unrelated to the stated scope and likely unintentional.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }; | ||
|
|
||
| comments.Comments.Add(comment); | ||
| post.CommentsCount = comments.Comments.Count; |
SEO analysis (HRINT-4790): - Post model: SeoMetaDescription, SeoKeywords, SeoLastAnalyzedAt fields - SeoHelper: JSON-LD BlogPosting structured data for public views - Public view: AI meta description with fallback, keywords meta tag, JSON-LD - Admin views: SEO analysis panel (read-only) in post details and edit - GenAI task on Posts collection generates SEO metadata via "ai-chat" connection Replace Akismet with GenAI spam check (HRINT-4788): - SpamCheckStatus enum (Pending, Valid, Spam) on PostComments.Comment - Comments start as Pending, visible only to poster (by cookie) or authenticated users - GenAI task evaluates pending comments, marks Valid or Spam - Valid: sets IsTrustedCommenter, triggers email notification - Spam: moves to Spam list, decrements CommentsCount, accumulates in daily SpamDigest - Remove Akismet dependency, AkismetService, TaskExecutor, AddCommentTask infrastructure - EmailSubscription worker consumes EmailCommands via RavenDB subscription - PostComments_CreationDate index filters out Pending comments - Admin UI: PENDING/SPAM labels on comments, SpamCheckStatus in MarkSpam/MarkHam Infrastructure: - All GenAI tasks use "ai-chat" connection string (configured manually in RavenDB Studio) - Document refresh enabled for @refresh-based future post triggers - "email-worker" data subscription expected in RavenDB Studio
Social media GenAI task: - Generates platform-tailored TwitterText and RedditTitle for published posts - Regenerates on post updates, skips historical backlog on initial deployment - Sets @refresh metadata for future posts to trigger posting at publish time Social posting subscription: - RavenDB subscription on Posts collection replaces FluentScheduler polling - Posts to Reddit (pending RedditSharp 2.0 migration) and Twitter (stub) - Checks: published, AI text ready, correct tags, not disabled Tag convention (stored as slugs): @social → post to all platforms @social/reddit → Reddit only @social/twitter → Twitter only @social/disable → kill all social posting Legacy "reddit" tag still accepted for backward compatibility Filter @-prefixed tags from all public views (post details, post list, related posts) Model: SocialMedia (TwitterText, RedditTitle, GeneratedAt, DisableAutoPublish) on Post
Demote h1-h6 sizes inside .text-wrapper so body headings are visually subordinate to the post title (h2 at 36px desktop / 24px mobile): h1 → 26px, h2 → 22px, h3 → 18px, h4+ → 14px
Both EmailSubscription and SocialPostingSubscription now call EnsureSubscriptionExists() before getting the worker. If the named subscription doesn't exist in RavenDB, it's created automatically. No more manual setup required in RavenDB Studio.
- email-worker: filter 'where SendTo != null', start from LastDocument - social-posting: filter 'where Social.GeneratedAt != null', start from LastDocument ChangeVector = "LastDocument" ensures the subscription only processes new documents, not the entire backlog. The query filters ensure the subscription only fires for documents that are actually ready to process.
- Subscriptions trigger on lack of @refresh metadata - refactoring spam-filter gen ai task
Fixes in spam filter UpdateScript: - Fix SpamDigest creation: load || default syntax was broken (comma operator), use explicit if/null check with put() passing metadata separately - Fix undefined blogName: load BlogConfig in Valid branch before referencing it EmailSubscription resolves SendTo from BlogConfig.OwnerEmail when not set on the command. The GenAI UpdateScript no longer sets SendTo — the email worker handles recipient resolution, keeping the UpdateScript simpler. Subscription query changed from 'SendTo != null' to 'Subject != null' since SendTo is no longer set by the UpdateScript.
- SpamDigest uses load(id) || {} pattern instead of if/else
- Remove BlogName from UpdateScript email commands — the EmailSubscription
resolves it from BlogConfig at send time (falls back to cmd.BlogName
for backward compat with existing documents)
- BlogConfig is loaded once per command for both SendTo and BlogName
Fix MarkHam in admin PostsController: the old code iterated all comments with IsSpam=true instead of just the selected ones. Now only processes the ham array (comments selected by the admin). Clean up subscription workers: replace async lambda + await Task.CompletedTask with synchronous lambda + return Task.CompletedTask.
GenAI tasks extracted to separate files: - Infrastructure/GenAiTasks/SpamFilterGenAiTask.cs - Infrastructure/GenAiTasks/SeoAnalysisGenAiTask.cs - Infrastructure/GenAiTasks/SocialMediaGenAiTask.cs Social posting subscription: - Set @refresh on future posts so subscription fires at publish time - Move Social.GeneratedAt and DisableAutoPublish checks to subscription query - Twitter posting via X API v2 with Bearer token from BlogConfig - Reddit posting logged as pending RedditSharp 2.0 migration Email templates: - Switch from inline HTML to Scriban 7.2.0 templates - Proper HTML email layout matching blog style (dark header #2c3e50) - NewComment template: post link, comment body, author info, action buttons - SpamDigest template: date summary, spam entries with preview, admin link Spam rejection for repeat offenders: - Commenter with >4 spam comments: return success but don't save comment - User sees "posted soon" message but comment is silently discarded BlogConfig: add TwitterBearerToken field + admin settings UI
RavenDB patch put() signature is put(id, doc) or put(id, doc, changeVector). The third argument must be a string (change vector), not a metadata object. Set metadata via @metadata property on the document itself instead. Fixes: "The change vector must be a string or null" error when creating SpamDigest and NewComment EmailCommand documents from the UpdateScript.
Post model: replace flat SeoMetaDescription/SeoKeywords/SeoLastAnalyzedAt
with Post.Seo (SeoMetadata) — matches the Social/Integration pattern.
GenAI UpdateScript writes this.Seo = { MetaDescription, Keywords, LastAnalyzedAt }.
AutoMapper bridges nested model to flat view model properties.
PostInput stays flat for form binding, mapped to SeoMetadata on save.
All three GenAI tasks now try Add first, and if the task already exists, fall back to Update (keeping the existing change vector). This ensures prompt/script changes are deployed on app restart without requiring manual deletion of the task in RavenDB Studio.
- Spam digest doc now sets Type/Subject/DigestDate (+ PostTitle per entry) so the email-worker subscription matches it and the digest template renders - EmailSubscription reads SmtpSettings from IConfiguration (Web.config is gone); sets host/port/credentials/SSL/from instead of a parameterless SmtpClient - Admin SEO fields are editable; LastAnalyzedAt round-trips via a hidden field so saving a post no longer wipes AI-generated SEO metadata - Spam-filter email script: derive PostSlug from title (Post has no Slug) and null-guard post before dereferencing ShowPostEvenIfPrivate - SocialPostingSubscription: post to Twitter asynchronously (drop .Result) and guard against empty Social.TwitterText - Remove no-op IsTrustedCommenter assignment in PostDetailsController - Remove orphan AkismetKey from BlogConfig/BlogConfigurationInput/importer - Subscription existence checks create on null state or DoesNotExist exception - GenAiTaskHelper.RegisterOrUpdate looks up the real task id via GetOngoingTaskInfoOperation before UpdateGenAiOperation (config.TaskId was 0), so task config changes propagate on startup; used by all three GenAI tasks
- Move NewComment/SpamDigest email bodies out of EmailSubscription into Infrastructure/EmailTemplates/*.html, embedded as resources and loaded (cached) by resource-name suffix at runtime - Remove redundant PostComments.Comment.IsSpam; spam state is tracked solely by SpamCheckStatus (Pending/Valid/Spam). Updated the spam-filter script, admin moderation, comment creation, the CreationDate index, the admin view model + Details.cshtml, and both import tools - Delete the dead Razor email path now replaced by EmailSubscription+Scriban: Views/MailTemplates/*, NewCommentEmailViewModel, SpamDigestEmailViewModel, RedditSubmissionFailedViewModel, EmailViewModelMapperProfile (+ registration). The Reddit artifacts were only referenced by the #if FALSE Reddit strategy. - Update AutoMapper config test accordingly
Design for re-enabling automatic Reddit posting under SocialPostingSubscription (RedditSharp 2.0, stateless via DuplicateLinkException) and owner failure emails for both Twitter and Reddit with full exception detail.
acad3f2 to
97d41fa
Compare
RedditSubredditsToSubmitToOnPublish is now List<string> instead of a comma-separated string, stored as a native JSON array on Blog/Config. Admin Settings edits it as one-per-line text (split at the controller). ParseSubreddits now trims/filters the list and null-guards BlogConfig. Migration: existing Blog/Config docs with a string value must be converted to an array in RavenDB Studio (see setup guide) before load.
EnsureSubscriptionExists and GetSubscriptionWorker made synchronous RavenDB calls in Start(), so app startup threw when RavenDB was unreachable — this crashed WebApplicationFactory-based integration tests (Connection refused) and would take down the app on a transient DB outage at boot. Move all RavenDB interaction (subscription creation + worker loop) inside the background Task.Run under the existing try/catch, for both the email-worker and social-posting subscriptions. Startup no longer blocks on or dies from RavenDB connectivity; workers come up on their own once the server responds.
- Drop the public share buttons (Tweet / Facebook / LinkedIn) from post Details and the post list. - Remove Facebook & Twitter commenter OAuth login: AddFacebook/AddTwitter wiring (Program.cs), their OAuth config (appsettings), the Facebook/Twitter OAuth key accessors (ConfigurationHelper), the buttons in the (commented) login form, and the two auth NuGet packages. Google/Microsoft login kept. - Remove the owner's Twitter/Facebook/LinkedIn profile links from Contact (GitHub/RSS/email kept), and drop FacebookLogin/LinkedInLogin from BlogConfig and the admin Settings form. Kept intentionally: the Twitter/Reddit auto-publish feature (sharing != auto- publish) and the twitter:card / OpenGraph meta tags (link previews), which is why BlogConfig.TwitterLogin remains.
The commenter OAuth sign-in was never actually used (the login UI was commented out). Remove the whole concept: - Delete SocialController (external login + callback). - Remove the Google/Microsoft OAuth provider registration from Program.cs (the admin cookie auth on the same builder is kept) and their config (appsettings Raccoon:OAuth) and the two auth NuGet packages. - Remove the OAuth key accessors + GetKeys from ConfigurationHelper. - Remove the commented-out "Sign in with" block from the comment form. - Drop the now-dead data model: Commenter.OpenId, PostViewModel .IsLoggedInCommenter, and the IsLoggedInCommenter/OpenId logic in PostDetailsController + the OpenId AutoMapper ignore. Cookie-based commenter recognition (CommenterKey) and CAPTCHA-bypass trust (IsTrustedCommenter) are unaffected.
The ContactMe section view is rendered directly as a partial from _Layout (sidebar); the SectionController.ContactMe() action was leftover from the old MVC child-action pattern and never on the render path. Removing it does not change what renders.
Keep the owner's Twitter/Facebook/LinkedIn profile links in the Contact sidebar (and their FacebookLogin/LinkedInLogin fields on BlogConfig + the admin Settings editors). This reverts only the profile-links slice of the social-sharing removal; the public share buttons and commenter social login stay removed.
Anonymous, non-trusted commenters always hit CAPTCHA validation, which returned "Captcha response not supplied." whenever Recaptcha:SiteKey/Secret were unset (no widget renders, so no token is posted) — blocking all commenting on any deployment without reCAPTCHA keys. Skip validation and skip rendering the widget/script when the keys aren't configured (mirrors the OAuth "only enable if configured" convention). When both keys are set, the checkbox and server-side verification work as before.
| comments.Comments.Add(comment); | ||
| post.CommentsCount = comments.Comments.Count; | ||
|
|
- Subscription workers (email-worker, social-posting) now retry with a delay instead of exiting on the first failure, so they self-heal when RavenDB becomes reachable (previously a startup-time outage disabled them until an app restart, despite the comment claiming otherwise). - EmailSubscription fails fast with a clear warning when SmtpSettings:From is not configured (previously MailMessage threw at send time for every command). - Escape HTML in the JSON-LD structured data (StringEscapeHandling.EscapeHtml) so a post field containing "</script>" can't break out of the script tag. - HTML-escape user-supplied fields in the NewComment and SpamDigest email templates (author, email, url, body, IP, user-agent, post title) to prevent HTML injection in notification emails. Not changed: CommentsCount includes Pending comments — a deliberate decision (pending comments are counted).
Summary
@socialtag conventionKey changes
SEO (HRINT-4790)
Postmodel:SeoMetaDescription,SeoKeywords,SeoLastAnalyzedAtSeoHelper: JSON-LD BlogPosting structured dataseo-analysistask on Posts collectionSpam check (HRINT-4788)
SpamCheckStatusenum (Pending, Valid, Spam) onPostComments.Commentspam-filtertask evaluates comments — Valid triggers email, Spam decrements count and accumulates in daily digestIsTrustedCommenteron valid comments (CAPTCHA bypass for returning commenters)EmailSubscriptionworker via RavenDB data subscriptionSocial media (HRINT-4789)
SocialMediamodel:TwitterText,RedditTitle,GeneratedAt,DisableAutoPublishsocial-mediatask generates text on post updates, skips historical backlogSocialPostingSubscriptionvia RavenDB subscription (replaces FluentScheduler polling)@refreshmetadata for future posts triggers posting at publish time@social(all),@social/reddit,@social/twitter,@social/disable@-prefixed tags filtered from all public viewsHeading fix
.text-wrapperso body headings sit below post titleManual setup required in RavenDB Studio
ai-chatAI connection string (chat-capable model)email-workerdata subscription (EmailCommands collection)social-postingdata subscription (Posts collection)Migration patch for existing comments
Run in RavenDB Studio to set SpamCheckStatus on existing comments:
Model compatibility
All changes are additive (new fields only). Old code ignores new fields; RavenDB preserves unknown JSON fields on load/save. Safe for rolling deployment.
Test plan
dotnet build)ai-chatconnection string in RavenDB Studioemail-workerandsocial-postingdata subscriptions@socialtag to a post and verify social text is generated@-prefixed tags are hidden from public views