Rename vlf-integrate.el to vlf-setup.el

when moving to next batch and be more precise with hexl-mode active.

README.org

@@ -1,28 +1,31 @@
 * View Large Files
 
 Emacs minor mode that allows viewing, editing, searching and comparing
-large files in batches.  Batch size can be adjusted on the fly and
+large files in batches, trading memory for processor time.  Batch size
-bounds the memory that is to be used for operations on the file.  This
+can be adjusted on the fly and bounds the memory that is to be used
-way multiple large files can be instantly and simultaneously accessed
+for operations on the file.  This way multiple large files (like
+terabytes or whatever) can be instantly and simultaneously accessed
 without swapping and degraded performance.
 
-This is development version of the GNU ELPA [[http://elpa.gnu.org/packages/vlf][vlf.el]] package.  Here's
+This is development version of the GNU ELPA [[http://elpa.gnu.org/packages/vlf][VLF]] package.  Here's what
-what it does in a nutshell:
+it offers in a nutshell:
 
-- regular expression search on whole file (in constant memory
+- automatic adjustment of batch size for optimal performance and
-  determined by current batch size)
+  responsiveness
-- chunk editing (if size has changed, saving is done in constant
+- regular expression search and replace over whole file
-  memory determined by current batch size)
+- [[http://www.emacswiki.org/emacs/OccurMode][Occur]] like indexing
-- occur like indexing
+- by batch [[http://www.emacswiki.org/emacs/EdiffMode][Ediff]] comparison
+- automatic scrolling of batches
+- chunk editing (save is immediate if size hasn't changed, done in
+  constant memory determined by current batch size otherwise)
 - options to jump to beginning, end or arbitrary file chunk
-- ability to jump/insert given number of batches at once
+- smooth integration with [[http://www.emacswiki.org/emacs/HexlMode][hexl-mode]]
+- works with [[http://www.emacswiki.org/emacs/TrampMode][TRAMP]] so accessing network files is fine and quick
 - newly added content is acknowledged if file has changed size
   meanwhile
-- automatic scrolling of batches
+- as it's a minor mode, font locking and functionality of the
-- as VLF is minor mode, font locking and functionality of the
+  respective major mode and other minor modes is also present
-  respective major mode is also present
+- can be added as option to automatically open large files
-- by batch Ediff comparison
-- VLF is added as an option when opening large files
 
 GNU Emacs 23 and 24 are supported.
 
@@ -34,76 +37,98 @@ M-x vlf PATH-TO-FILE
 
 Emacs' Unicode support is leveraged so you'll not see bare bytes but
 characters decoded as if file is normally opened.  This holds for
-editing, search and indexing.
+editing, search, indexing and comparison.
 
 ** 32-bit GNU Emacs
 
 Regular Emacs integers are used, so if you use 32-bit Emacs without
-bignum support, VLF will not work with files over 512 MB (maximum
+bignum support, *VLF* will not work with files over 512 MB (maximum
 integer value).
 
-** Memory control
-
-*vlf-batch-size* bounds the memory used for all operations.
-
 * Detail usage
 
 ** Applicability
 
 To have *vlf* offered as choice when opening large files:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC emacs-lisp
-(require 'vlf-integrate)
+  (require 'vlf-setup)
-#+END_EXAMPLE
+#+END_SRC
 
-You can control when vlf-mode is invoked or offered with the
+You can control when *vlf-mode* is invoked or offered with the
-*vlf-application* customization option.  By default it will offer VLF
+*vlf-application* customization option.  By default it will offer
-when opening large files.  There are also options to never use it (you
+*VLF* when opening large files.  There are also options to never use
-can still call *vlf* command explicitly); to use it without asking for
+it (you can still call *vlf* command explicitly); to use it without
-large files or to invoke it on all files.  Here's example setup such
+asking for large files or to invoke it on all files.  Here's example
-that vlf-mode automatically launches for large files:
+setup such that *vlf-mode* automatically launches for large files:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC emacs-lisp
-(custom-set-variables
+  (custom-set-variables
- '(vlf-application 'dont-ask))
+   '(vlf-application 'dont-ask))
-#+END_EXAMPLE
+#+END_SRC
 
 *** Disable for specific mode
 
-To disable automatic usage of VLF for a major mode, add it to
+To disable automatic usage of *VLF* for a major mode, add it to
 *vlf-forbidden-modes-list*.
 
 *** Disable for specific function
 
-To disable automatic usage of VLF for a function, for example named
+To disable automatic usage of *VLF* for a function, for example named
 *func* defined in file *file.el*:
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC emacs-lisp
-(vlf-disable-for-function func "file")
+  (vlf-disable-for-function func "file")
-#+END_EXAMPLE
+#+END_SRC
 
 ** Keymap
 
-All VLF operations are grouped under the *C-c C-v* prefix by default.
+All *VLF* operations are grouped under the *C-c C-v* prefix by
-Here's example how to add another prefix (*C-x v*):
+default.  Here's example how to add another prefix (*C-x v*):
 
-#+BEGIN_EXAMPLE
+#+BEGIN_SRC emacs-lisp
-(eval-after-load "vlf"
+  (eval-after-load "vlf"
-  '(define-key vlf-prefix-map "\C-xv" vlf-mode-map))
+    '(define-key vlf-prefix-map "\C-xv" vlf-mode-map))
-#+END_EXAMPLE
+#+END_SRC
 
-** Control batch size
+** Overall position indicators
+
+To see which part of the file is currently visited and how many
+batches there are in overall (using the current batch size), look at
+the VLF section in the mode line, file size is also there.
+
+** Batch size control
+
+By default *VLF* gathers statistics over how primitive operations
+perform over file and gradually adjusts batch size for better user
+experience.  Operations involving multiple batches are tuned more
+adventurously.  Overall the more jumping around, searching, indexing,
+the better performance should get.
+
+The *vlf-tune-max* option specifies maximum size in bytes a batch
+could eventually get while tuning.
+
+Profiling and tuning can be disabled by:
+
+#+BEGIN_SRC emacs-lisp
+  (custom-set-variables
+     '(vlf-tune-enabled nil))
+#+END_SRC
+
+Or set *vlf-tune-enabled* to '*stats* to profile but not change batch
+size.
 
 Use *M-x vlf-set-batch-size* to change batch size and update chunk
-immediately.
+immediately.  Default size offered is the best according to tune
+statistics so far.
 
 *C-c C-v +* and *C-c C-v -* control current batch size by factors
 of 2.
 
 ** Move around
 
-Scrolling automatically triggers moving to previous or next chunk at
+Scrolling automatically triggers move to previous or next chunk at the
-the beginning or end respectively of the current one.
+beginning or end respectively of the current one.
 
 *C-c C-v n* and *C-c C-v p* move batch by batch.  With positive
 prefix argument they move prefix number of batches.  With negative -
@@ -114,42 +139,41 @@ append prefix number of batches.
 *C-c C-v [* and *C-c C-v ]* take you to the beginning and end of file
 respectively.
 
-*C-c C-v j* jumps to given chunk.  To see where you are in file and
+*C-c C-v j* jumps to particular batch number.
-how many chunks there are (using the current batch size), look at the
-parenthesized part of the buffer name, batch size is also indicated at
-the end.
 
 ** Follow point
 
 Continuous chunk recenter around point in current buffer can be
 toggled with *C-c C-v f*.
 
-** Search whole file
+** Search and/or replace whole file
 
 *C-c C-v s* and *C-c C-v r* search forward and backward respectively
-over the whole file.  This is done batch by batch so if you have
+over the whole file, batch by batch.  *C-c C-v %* does search and
-really huge file - you'd better set somewhat bigger batch size
+query replace saving intermediate changes.
-beforehand.
 
 ** Occur over whole file
 
-*C-c C-v o* builds index for given regular expression just like
+*C-c C-v o* builds index over whole file for given regular expression
-*M-x occur*.  It does this batch by batch over the whole file.  Note
+just like *M-x occur*.  Note that even if you prematurely stop it with
-that even if you prematurely stop it with *C-g*, it will still show
+*C-g*, it will still show what's found so far.
-index of what's found so far.
+
+Result buffer uses *vlf-occur-mode* which allows to optionally open
+new *VLF* buffer on jump to match (using *C-u* before hitting RET or
+*o*), thus having multiple simultaneous views of the same file.  Also
+results can be serialized to file for later reuse.
 
 ** Jump to line
 
-*C-c C-v l* jumps to given line in file.  This is done by searching
+*C-c C-v l* jumps to given line in file.  With negative argument,
-from the beginning, so again the bigger current batch size, the
+lines are counted from the end of file.
-quicker.  With negative argument, lines are counted from the end of
-file.
 
 ** Edit and save
 
 If editing doesn't change size of the chunk, only this chunk is saved.
-Otherwise the remaining part of the file is adjusted batch by batch,
+Otherwise the remaining part of the file is adjusted batch by batch.
-so again you'd better have bigger current batch size.
+*vlf-save-in-place* customization option controls if temporary file
+should be used in such case.
 
 ** By batch Ediff
 
@@ -159,3 +183,18 @@ or taken from the first buffer in case of buffers).  Moving after the
 last difference in current chunk searches for following one with
 difference.  The other way around if looking for difference before the
 first one.
+
+* Extend
+
+** Move hooks
+
+A couple of hooks are run whenever updating chunk:
+*vlf-before-chunk-update-hook* and *vlf-after-chunk-update-hook*.
+
+** Batch move hooks
+
+Some operations may trigger multiple chunk moves.  There are a couple
+of hooks that run in such cases: *vlf-before-batch-functions* and
+*vlf-after-batch-functions*.  They are passed one argument which
+specifies type of operation that runs.  Possible values are the
+symbols: *write*, *ediff*, *occur*, *search* and *goto-line*.

vlf-base.el

@@ -1,6 +1,6 @@
 ;;; vlf-base.el --- VLF primitive operations  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2014 Free Software Foundation, Inc.
+;; Copyright (C) 2014-2015 Free Software Foundation, Inc.
 
 ;; Keywords: large files, chunk
 ;; Author: Andrey Kotlarski <m00naticus@gmail.com>
@@ -27,16 +27,15 @@
 
 ;;; Code:
 
-(defgroup vlf nil
+(require 'vlf-tune)
-  "View Large Files in Emacs."
-  :prefix "vlf-"
-  :group 'files)
 
-(defcustom vlf-batch-size 1024
+(defcustom vlf-before-chunk-update-hook nil
-  "Defines how large each batch of file data is (in bytes)."
+  "Hook that runs before chunk update."
-  :group 'vlf
+  :group 'vlf :type 'hook)
-  :type 'integer)
+
-(put 'vlf-batch-size 'permanent-local t)
+(defcustom vlf-after-chunk-update-hook nil
+  "Hook that runs after chunk update."
+  :group 'vlf :type 'hook)
 
 ;;; Keep track of file position.
 (defvar vlf-start-pos 0
@@ -48,9 +47,7 @@
 (make-variable-buffer-local 'vlf-end-pos)
 (put 'vlf-end-pos 'permanent-local t)
 
-(defvar vlf-file-size 0 "Total size of presented file.")
+(defvar hexl-bits)
-(make-variable-buffer-local 'vlf-file-size)
-(put 'vlf-file-size 'permanent-local t)
 
 (defconst vlf-sample-size 24
   "Minimal number of bytes that can be properly decoded.")
@@ -59,11 +56,13 @@
   "Get size in bytes of FILE."
   (or (nth 7 (file-attributes file)) 0))
 
-(defun vlf-verify-size (&optional update-visited-time)
+(defun vlf-verify-size (&optional update-visited-time file)
   "Update file size information if necessary and visited file time.
-If non-nil, UPDATE-VISITED-TIME."
+If non-nil, UPDATE-VISITED-TIME.
+FILE if given is filename to be used, otherwise `buffer-file-truename'."
   (unless (verify-visited-file-modtime (current-buffer))
-    (setq vlf-file-size (vlf-get-file-size buffer-file-truename))
+    (setq vlf-file-size (vlf-get-file-size (or file
+                                               buffer-file-truename)))
     (if update-visited-time
         (set-visited-file-modtime))))
 
@@ -72,15 +71,6 @@ If non-nil, UPDATE-VISITED-TIME."
     "Print FILE-SIZE in MB."
     (format "%.3fMB" (/ file-size 1048576.0))))
 
-(defun vlf-update-buffer-name ()
-  "Update the current buffer name."
-  (rename-buffer (format "%s(%d/%d)[%s]"
-                         (file-name-nondirectory buffer-file-name)
-                         (/ vlf-end-pos vlf-batch-size)
-                         (/ vlf-file-size vlf-batch-size)
-                         (file-size-human-readable vlf-batch-size))
-                 t))
-
 (defmacro vlf-with-undo-disabled (&rest body)
   "Execute BODY with temporarily disabled undo."
   `(let ((undo-list buffer-undo-list))
@@ -88,33 +78,30 @@ If non-nil, UPDATE-VISITED-TIME."
      (unwind-protect (progn ,@body)
        (setq buffer-undo-list undo-list))))
 
-(defun vlf-move-to-chunk (start end &optional minimal)
+(defun vlf-move-to-chunk (start end)
   "Move to chunk enclosed by START END bytes.
-When given MINIMAL flag, skip non important operations.
 If same as current chunk is requested, do nothing.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
   (vlf-verify-size)
-  (cond ((or (<= end start) (<= end 0)
+  (if (or (<= end start) (<= end 0)
-             (<= vlf-file-size start))
+          (<= vlf-file-size start))
-         (when (or (not (buffer-modified-p))
+      (when (or (not (buffer-modified-p))
-                   (y-or-n-p "Chunk modified, are you sure? "))
+                (y-or-n-p "Chunk modified, are you sure? "))
-           (erase-buffer)
+        (erase-buffer)
-           (set-buffer-modified-p nil)
+        (set-buffer-modified-p nil)
-           (let ((place (if (<= vlf-file-size start)
+        (let ((place (if (<= vlf-file-size start)
-                            vlf-file-size
+                         vlf-file-size
-                          0)))
+                       0)))
-             (setq vlf-start-pos place
+          (setq vlf-start-pos place
-                   vlf-end-pos place)
+                vlf-end-pos place)
-             (if (not minimal)
+          (cons (- start place) (- place end))))
-                 (vlf-update-buffer-name))
+    (if (derived-mode-p 'hexl-mode)
-             (cons (- start place) (- place end)))))
+        (setq start (- start (mod start hexl-bits))
-        ((or (/= start vlf-start-pos)
+              end (+ end (- hexl-bits (mod end hexl-bits)))))
-             (/= end vlf-end-pos))
+    (if (or (/= start vlf-start-pos)
-         (let ((shifts (vlf-move-to-chunk-1 start end)))
+            (/= end vlf-end-pos))
-           (and shifts (not minimal)
+        (vlf-move-to-chunk-1 start end))))
-                (vlf-update-buffer-name))
-           shifts))))
 
 (defun vlf-move-to-chunk-1 (start end)
   "Move to chunk enclosed by START END keeping as much edits if any.
@@ -124,116 +111,167 @@ bytes added to the end."
   (let* ((modified (buffer-modified-p))
          (start (max 0 start))
          (end (min end vlf-file-size))
+         (hexl (derived-mode-p 'hexl-mode))
+         restore-hexl hexl-undo-list
          (edit-end (if modified
-                       (+ vlf-start-pos
+                       (progn
-                          (length (encode-coding-region
+                         (when hexl
-                                   (point-min) (point-max)
+                           (setq restore-hexl t
-                                   buffer-file-coding-system t)))
+                                 hexl-undo-list buffer-undo-list
-                     vlf-end-pos)))
+                                 buffer-undo-list t)
-    (cond
+                           (vlf-tune-dehexlify))
-     ((or (< edit-end start) (< end vlf-start-pos)
+                         (+ vlf-start-pos
-          (not (verify-visited-file-modtime (current-buffer))))
+                            (vlf-tune-encode-length (point-min)
-      (when (or (not modified)
+                                                    (point-max))))
-                (y-or-n-p "Chunk modified, are you sure? ")) ;full chunk renewal
+                     vlf-end-pos))
-        (set-buffer-modified-p nil)
+         (shifts
-        (vlf-move-to-chunk-2 start end)))
+          (cond
-     ((and (= start vlf-start-pos) (= end edit-end))
+           ((and hexl (not modified)) (vlf-move-to-chunk-2 start end))
-      (or modified (vlf-move-to-chunk-2 start end)))
+           ((or (< edit-end start) (< end vlf-start-pos)
-     ((or (and (<= start vlf-start-pos) (<= edit-end end))
+                (not (verify-visited-file-modtime (current-buffer))))
-          (not modified)
+            (when (or (not modified)
-          (y-or-n-p "Chunk modified, are you sure? "))
+                      (y-or-n-p "Chunk modified, are you sure? ")) ;full chunk renewal
-      (let ((shift-start 0)
+              (set-buffer-modified-p nil)
-            (shift-end 0))
+              (if (consp hexl-undo-list)
-        (let ((pos (+ (position-bytes (point)) vlf-start-pos))
+                  (setq hexl-undo-list nil))
-              (inhibit-read-only t))
+              (vlf-move-to-chunk-2 start end)))
-          (cond ((= end vlf-start-pos)
+           ((and (= start vlf-start-pos) (= end edit-end))
-                 (or (eq buffer-undo-list t)
+            (unless modified
-                     (setq buffer-undo-list nil))
+              (if (consp hexl-undo-list)
-                 (vlf-with-undo-disabled (erase-buffer))
+                  (setq hexl-undo-list nil))
-                 (setq modified nil))
+              (vlf-move-to-chunk-2 start end)))
-                ((< end edit-end)
+           ((and (not modified)
-                 (setq end (car (vlf-delete-region
+                 (not (consp buffer-undo-list)))
-                                 (point-min) vlf-start-pos edit-end
+            (vlf-move-to-chunk-2 start end))
-                                 end (min (or (byte-to-position
+           ((or (not modified)
-                                               (- end vlf-start-pos))
+                (and (<= start vlf-start-pos) (<= edit-end end))
-                                              (point-min))
+                (y-or-n-p "Chunk modified, are you sure? "))
-                                          (point-max))
+            (run-hooks 'vlf-before-chunk-update-hook)
-                                 nil))))
+            (when (and hexl (not restore-hexl))
-                ((< edit-end end)
+              (if (consp buffer-undo-list)
-                 (vlf-with-undo-disabled
+                  (setq buffer-undo-list nil))
-                  (setq shift-end (cdr (vlf-insert-file-contents
+              (vlf-tune-dehexlify))
-                                        vlf-end-pos end nil t
+            (let ((shift-start 0)
-                                        (point-max)))))))
+                  (shift-end 0))
-          (setq vlf-end-pos (+ end shift-end))
+              (let ((pos (+ (position-bytes (point)) vlf-start-pos))
-          (cond ((= start edit-end)
+                    (inhibit-read-only t))
-                 (or (eq buffer-undo-list t)
+                (cond ((= end vlf-start-pos)
-                     (setq buffer-undo-list nil))
+                       (or (eq buffer-undo-list t)
-                 (vlf-with-undo-disabled
+                           (setq buffer-undo-list nil))
-                  (delete-region (point-min) (point)))
+                       (vlf-with-undo-disabled (erase-buffer))
-                 (setq modified nil))
+                       (setq modified nil))
-                ((< vlf-start-pos start)
+                      ((< end edit-end)
-                 (let ((del-info (vlf-delete-region
+                       (setq end (car (vlf-delete-region
-                                  (point-min) vlf-start-pos
+                                       (point-min) vlf-start-pos
-                                  vlf-end-pos start
+                                       edit-end end
-                                  (min (or (byte-to-position
+                                       (min (or (byte-to-position
-                                            (- start vlf-start-pos))
+                                                 (- end vlf-start-pos))
-                                           (point))
+                                                (point-min))
-                                       (point-max)) t)))
+                                            (point-max))
-                   (setq start (car del-info))
+                                       nil))))
-                   (vlf-shift-undo-list (- (point-min)
+                      ((< edit-end end)
-                                           (cdr del-info)))))
+                       (vlf-with-undo-disabled
-                ((< start vlf-start-pos)
+                        (setq shift-end (cdr (vlf-insert-file-contents
-                 (let ((edit-end-pos (point-max)))
+                                              vlf-end-pos end nil t
-                   (vlf-with-undo-disabled
+                                              (point-max)))))))
-                    (setq shift-start (car (vlf-insert-file-contents
+                (setq vlf-end-pos (+ end shift-end))
-                                            start vlf-start-pos t nil
+                (cond ((= start edit-end)
-                                            edit-end-pos)))
+                       (or (eq buffer-undo-list t)
-                    (goto-char (point-min))
+                           (setq buffer-undo-list nil))
-                    (insert (delete-and-extract-region
+                       (vlf-with-undo-disabled
-                             edit-end-pos (point-max))))
+                        (delete-region (point-min) (point)))
-                   (vlf-shift-undo-list (- (point-max)
+                       (setq modified nil))
-                                           edit-end-pos)))))
+                      ((< vlf-start-pos start)
-          (setq start (- start shift-start))
+                       (let ((del-info (vlf-delete-region
-          (goto-char (or (byte-to-position (- pos start))
+                                        (point-min) vlf-start-pos
-                         (byte-to-position (- pos vlf-start-pos))
+                                        vlf-end-pos start
-                         (point-max)))
+                                        (min (or
-          (setq vlf-start-pos start))
+                                              (byte-to-position
-        (set-buffer-modified-p modified)
+                                               (- start vlf-start-pos))
-        (set-visited-file-modtime)
+                                              (point))
-        (cons shift-start shift-end))))))
+                                             (point-max)) t)))
+                         (setq start (car del-info))
+                         (vlf-shift-undo-list (- (point-min)
+                                                 (cdr del-info)))))
+                      ((< start vlf-start-pos)
+                       (let ((edit-end-pos (point-max)))
+                         (vlf-with-undo-disabled
+                          (setq shift-start (car
+                                             (vlf-insert-file-contents
+                                              start vlf-start-pos t nil
+                                              edit-end-pos)))
+                          (goto-char (point-min))
+                          (insert (delete-and-extract-region
+                                   edit-end-pos (point-max))))
+                         (vlf-shift-undo-list (- (point-max)
+                                                 edit-end-pos)))))
+                (setq start (- start shift-start))
+                (goto-char (or (byte-to-position (- pos start))
+                               (byte-to-position (- pos vlf-start-pos))
+                               (point-max)))
+                (setq vlf-start-pos start))
+              (set-buffer-modified-p modified)
+              (set-visited-file-modtime)
+              (when hexl
+                (vlf-tune-hexlify)
+                (setq restore-hexl nil))
+              (run-hooks 'vlf-after-chunk-update-hook)
+              (cons shift-start shift-end))))))
+    (when restore-hexl
+      (vlf-tune-hexlify)
+      (setq buffer-undo-list hexl-undo-list))
+    shifts))
 
 (defun vlf-move-to-chunk-2 (start end)
   "Unconditionally move to chunk enclosed by START END bytes.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
-  (vlf-verify-size t)
+  (run-hooks 'vlf-before-chunk-update-hook)
-  (setq vlf-start-pos (max 0 start)
+  (let ((adjust-start t)
-        vlf-end-pos (min end vlf-file-size))
+        (adjust-end t)
-  (let (shifts)
+        (is-hexl (derived-mode-p 'hexl-mode)))
-    (let ((inhibit-read-only t)
+    (and (not is-hexl)
-          (pos (position-bytes (point))))
+         (verify-visited-file-modtime (current-buffer))
-      (vlf-with-undo-disabled
+         (setq adjust-start (and (/= start vlf-start-pos)
-       (erase-buffer)
+                                 (/= start vlf-end-pos))
-       (setq shifts (vlf-insert-file-contents vlf-start-pos
+               adjust-end (and (/= end vlf-start-pos)
-                                              vlf-end-pos t t)
+                               (/= end vlf-end-pos))))
-             vlf-start-pos (- vlf-start-pos (car shifts))
+    (vlf-verify-size t)
-             vlf-end-pos (+ vlf-end-pos (cdr shifts)))
+    (setq vlf-start-pos (max 0 start)
-       (goto-char (or (byte-to-position (+ pos (car shifts)))
+          vlf-end-pos (min end vlf-file-size))
-                      (point-max)))))
+    (let ((shifts '(0 . 0)))
-    (set-buffer-modified-p nil)
+      (let ((inhibit-read-only t)
-    (setq buffer-undo-list nil)
+            (pos (position-bytes (point))))
-    shifts))
+        (vlf-with-undo-disabled
+         (erase-buffer)
+         (if is-hexl
+             (progn (vlf-tune-insert-file-contents-literally
+                     vlf-start-pos vlf-end-pos)
+                    (vlf-tune-hexlify))
+           (setq shifts (vlf-insert-file-contents vlf-start-pos
+                                                  vlf-end-pos
+                                                  adjust-start
+                                                  adjust-end)
+                 vlf-start-pos (- vlf-start-pos (car shifts))
+                 vlf-end-pos (+ vlf-end-pos (cdr shifts)))))
+        (goto-char (or (byte-to-position (+ pos (car shifts)))
+                       (point-max))))
+      (set-buffer-modified-p nil)
+      (or (eq buffer-undo-list t)
+          (setq buffer-undo-list nil))
+      (run-hooks 'vlf-after-chunk-update-hook)
+      shifts)))
 
 (defun vlf-insert-file-contents (start end adjust-start adjust-end
                                        &optional position)
   "Adjust chunk at absolute START to END till content can be\
-properly decoded.  ADJUST-START determines if trying to prepend bytes\
+properly decoded.  ADJUST-START determines if trying to prepend bytes
- to the beginning, ADJUST-END - append to the end.
+to the beginning, ADJUST-END - append to the end.
 Use buffer POSITION as start if given.
 Return number of bytes moved back for proper decoding and number of
 bytes added to the end."
   (setq adjust-start (and adjust-start (not (zerop start)))
-        adjust-end (and adjust-end (< end vlf-file-size))
+        adjust-end (and adjust-end (/= end vlf-file-size))
         position (or position (point-min)))
   (goto-char position)
   (let ((shift-start 0)
@@ -245,7 +283,7 @@ bytes added to the end."
         (setq shift-start (vlf-adjust-start start safe-end position
                                             adjust-end)
               start (- start shift-start))
-      (vlf-insert-file-contents-1 start safe-end position))
+      (vlf-insert-file-contents-1 start safe-end))
     (if adjust-end
         (setq shift-end (- (car (vlf-delete-region position start
                                                    safe-end end
@@ -254,23 +292,9 @@ bytes added to the end."
                            end)))
     (cons shift-start shift-end)))
 
-(defun vlf-insert-file-contents-1 (start end position)
+(defun vlf-insert-file-contents-1 (start end)
-  "Extract decoded file bytes START to END at POSITION."
+  "Extract decoded file bytes START to END."
-  (let ((coding buffer-file-coding-system))
+  (vlf-tune-insert-file-contents start end))
-    (insert-file-contents-literally buffer-file-name nil start end)
-    (let ((coding-system-for-read coding))
-      (decode-coding-inserted-region position (point-max)
-                                     buffer-file-name nil start end)))
-  (when (eq (detect-coding-region position (min (+ position
-                                                   vlf-sample-size)
-                                                (point-max)) t)
-            'no-conversion)
-    (delete-region position (point-max))
-    (insert-file-contents-literally buffer-file-name nil start end)
-    (let ((coding-system-for-read nil))
-      (decode-coding-inserted-region position (point-max)
-                                     buffer-file-name nil start end)))
-  (setq buffer-file-coding-system last-coding-system-used))
 
 (defun vlf-adjust-start (start end position adjust-end)
   "Adjust chunk beginning at absolute START to END till content can\
@@ -283,8 +307,8 @@ Return number of bytes moved back for proper decoding."
          (strict (or (= sample-end vlf-file-size)
                      (and (not adjust-end) (= sample-end end))))
          (shift 0))
-    (while (and (progn (vlf-insert-file-contents-1
+    (while (and (progn (insert-file-contents buffer-file-name
-                        safe-start sample-end position)
+                                             nil safe-start sample-end)
                        (not (zerop safe-start)))
                 (< shift 3)
                 (let ((diff (- chunk-size
@@ -304,7 +328,7 @@ Return number of bytes moved back for proper decoding."
                                              position t 'start)))
     (unless (= sample-end end)
       (delete-region position (point-max))
-      (vlf-insert-file-contents-1 safe-start end position))
+      (vlf-insert-file-contents-1 safe-start end))
     (- start safe-start)))
 
 (defun vlf-delete-region (position start end border cut-point from-start
@@ -322,12 +346,10 @@ which deletion was performed."
                               (eq encode-direction 'end)
                             (< (- end border) (- border start))))
          (dist (if encode-from-end
-                   (- end (length (encode-coding-region
+                   (- end (vlf-tune-encode-length cut-point
-                                   cut-point (point-max)
+                                                  (point-max)))
-                                   buffer-file-coding-system t)))
+                 (+ start (vlf-tune-encode-length position
-                 (+ start (length (encode-coding-region
+                                                  cut-point))))
-                                   position cut-point
-                                   buffer-file-coding-system t)))))
          (len 0))
     (if (< border dist)
         (while (< border dist)
@@ -353,9 +375,17 @@ which deletion was performed."
        (delete-region cut-point (point-max))))
     (cons dist (1+ cut-point))))
 
+(defun vlf-byte-position (point)
+  "Determine global byte position of POINT."
+  (let ((pmax (point-max)))
+    (if (< (/ pmax 2) point)
+        (- vlf-end-pos (vlf-tune-encode-length (min (1+ point) pmax)
+                                               pmax))
+      (+ vlf-start-pos (vlf-tune-encode-length (point-min) point)))))
+
 (defun vlf-shift-undo-list (n)
   "Shift undo list element regions by N."
-  (or (eq buffer-undo-list t)
+  (or (null buffer-undo-list) (eq buffer-undo-list t)
       (setq buffer-undo-list
             (nreverse
              (let ((min (point-min))

vlf-ediff.el

@@ -1,4 +1,4 @@
-;;; vlf-ediff.el --- VLF ediff functionality
+;;; vlf-ediff.el --- VLF ediff functionality  -*- lexical-binding: t -*-
 
 ;; Copyright (C) 2014 Free Software Foundation, Inc.
 
@@ -34,7 +34,8 @@
   "If non nil, specifies that ediff is done over VLF buffers.")
 (make-variable-buffer-local 'vlf-ediff-session)
 
-;;;###autoload
+(defvar tramp-verbose)
+
 (defun vlf-ediff-buffers (buffer-A buffer-B)
   "Run batch by batch ediff over VLF buffers BUFFER-A and BUFFER-B.
 Batch size is determined by the size in BUFFER-A.
@@ -93,10 +94,10 @@ respectively of difference list, runs ediff over the adjacent chunks."
                                       dir-B)))
                                    (ediff-get-default-file-name f 1)))
            (read-number "Batch size (in bytes): " vlf-batch-size))))
-  (let ((buffer-A (vlf file-A)))
+  (let ((buffer-A (vlf file-A t)))
     (set-buffer buffer-A)
     (vlf-set-batch-size batch-size)
-    (let ((buffer-B (vlf file-B)))
+    (let ((buffer-B (vlf file-B t)))
       (vlf-ediff-buffers buffer-A buffer-B))))
 
 (defadvice ediff-next-difference (around vlf-ediff-next-difference
@@ -142,11 +143,11 @@ beginning of difference list."
 
 (defun vlf-next-chunk ()
   "Move to next chunk."
-  (vlf-move-to-chunk vlf-end-pos (+ vlf-end-pos vlf-batch-size) t))
+  (vlf-move-to-chunk vlf-end-pos (+ vlf-end-pos vlf-batch-size)))
 
 (defun vlf-prev-chunk ()
   "Move to previous chunk."
-  (vlf-move-to-chunk (- vlf-start-pos vlf-batch-size) vlf-start-pos t))
+  (vlf-move-to-chunk (- vlf-start-pos vlf-batch-size) vlf-start-pos))
 
 (defun vlf-ediff-next (buffer-A buffer-B ediff-buffer
                                 &optional next-func)
@@ -154,18 +155,23 @@ beginning of difference list."
 governed by EDIFF-BUFFER.  NEXT-FUNC is used to jump to the next
 logical chunks in case there is no difference at the current ones."
   (set-buffer buffer-A)
+  (run-hook-with-args 'vlf-before-batch-functions 'ediff)
   (setq buffer-A (current-buffer)) ;names change, so reference by buffer object
   (let ((end-A (= vlf-start-pos vlf-end-pos))
         (chunk-A (cons vlf-start-pos vlf-end-pos))
         (point-max-A (point-max))
         (font-lock-A font-lock-mode)
         (min-file-size vlf-file-size)
-        (forward-p (eq next-func 'vlf-next-chunk)))
+        (forward-p (eq next-func 'vlf-next-chunk))
+        (is-hexl (derived-mode-p 'hexl-mode)))
     (font-lock-mode 0)
     (set-buffer buffer-B)
+    (run-hook-with-args 'vlf-before-batch-functions 'ediff)
     (setq buffer-B (current-buffer)
-          min-file-size (min min-file-size vlf-file-size))
+          min-file-size (min min-file-size vlf-file-size)
-    (let ((tramp-verbose (min 2 tramp-verbose))
+          is-hexl (or is-hexl (derived-mode-p 'hexl-mode)))
+    (let ((tramp-verbose (if (boundp 'tramp-verbose)
+                             (min tramp-verbose 1)))
           (end-B (= vlf-start-pos vlf-end-pos))
           (chunk-B (cons vlf-start-pos vlf-end-pos))
           (font-lock-B font-lock-mode)
@@ -184,7 +190,7 @@ logical chunks in case there is no difference at the current ones."
                                     buffer-B (point-min) (point-max)))
                             (with-current-buffer ediff-buffer
                               (ediff-update-diffs)
-                              (and (not end-A) (not end-B)
+                              (and (not end-A) (not end-B) (not is-hexl)
                                    (vlf-ediff-refine buffer-A
                                                      buffer-B))
                               (zerop ediff-number-of-differences))))
@@ -199,10 +205,7 @@ logical chunks in case there is no difference at the current ones."
                                           (- vlf-file-size
                                              vlf-start-pos))))
             (progress-reporter-done reporter)
-            (if (or (not end-A) (not end-B))
+            (when (and end-A end-B)
-                (progn (vlf-update-buffer-name)
-                       (set-buffer buffer-A)
-                       (vlf-update-buffer-name))
               (if forward-p
                   (let ((max-file-size vlf-file-size))
                     (vlf-move-to-chunk (- max-file-size vlf-batch-size)
@@ -213,21 +216,16 @@ logical chunks in case there is no difference at the current ones."
                     (vlf-move-to-chunk (- max-file-size
                                           vlf-batch-size)
                                        max-file-size))
-                (vlf-beginning-of-file)
+                (vlf-move-to-batch 0)
                 (set-buffer buffer-A)
-                (vlf-beginning-of-file))
+                (vlf-move-to-batch 0))
               (set-buffer ediff-buffer)
               (ediff-update-diffs)
-              (if (or (not forward-p)
+              (or is-hexl
-                      (and (not end-A) (not end-B)))
+                  (if (or (not forward-p)
-                  (vlf-ediff-refine buffer-A buffer-B)))
+                          (and (not end-A) (not end-B)))
+                      (vlf-ediff-refine buffer-A buffer-B))))
             (setq done t))
-        (when font-lock-A
-          (set-buffer buffer-A)
-          (font-lock-mode 1))
-        (when font-lock-B
-          (set-buffer buffer-B)
-          (font-lock-mode 1))
         (unless done
           (set-buffer buffer-A)
           (set-buffer-modified-p nil)
@@ -237,7 +235,14 @@ logical chunks in case there is no difference at the current ones."
           (vlf-move-to-chunk (car chunk-B) (cdr chunk-B))
           (set-buffer ediff-buffer)
           (ediff-update-diffs)
-          (vlf-ediff-refine buffer-A buffer-B))))))
+          (or is-hexl
+              (vlf-ediff-refine buffer-A buffer-B)))
+        (set-buffer buffer-A)
+        (if font-lock-A (font-lock-mode 1))
+        (run-hook-with-args 'vlf-after-batch-functions 'ediff)
+        (set-buffer buffer-B)
+        (if font-lock-B (font-lock-mode 1))
+        (run-hook-with-args 'vlf-after-batch-functions 'ediff)))))
 
 (defun vlf-ediff-refine (buffer-A buffer-B)
   "Try to minimize differences between BUFFER-A and BUFFER-B.

vlf-occur.el

@@ -29,6 +29,24 @@
 
 (require 'vlf)
 
+(defvar vlf-occur-vlf-file nil "VLF file that is searched.")
+(make-variable-buffer-local 'vlf-occur-vlf-file)
+
+(defvar vlf-occur-vlf-buffer nil "VLF buffer that is scanned.")
+(make-variable-buffer-local 'vlf-occur-vlf-buffer)
+
+(defvar vlf-occur-regexp)
+(make-variable-buffer-local 'vlf-occur-regexp)
+
+(defvar vlf-occur-hexl nil "Is `hexl-mode' active?")
+(make-variable-buffer-local 'vlf-occur-hexl)
+
+(defvar vlf-occur-lines 0 "Number of lines scanned by `vlf-occur'.")
+(make-variable-buffer-local 'vlf-occur-lines)
+
+(defvar tramp-verbose)
+(defvar hexl-bits)
+
 (defvar vlf-occur-mode-map
   (let ((map (make-sparse-keymap)))
     (define-key map "n" 'vlf-occur-next-match)
@@ -37,16 +55,18 @@
     (define-key map "\M-\r" 'vlf-occur-visit-new-buffer)
     (define-key map [mouse-1] 'vlf-occur-visit)
     (define-key map "o" 'vlf-occur-show)
+    (define-key map [remap save-buffer] 'vlf-occur-save)
     map)
   "Keymap for command `vlf-occur-mode'.")
 
 (define-derived-mode vlf-occur-mode special-mode "VLF[occur]"
-  "Major mode for showing occur matches of VLF opened files.")
+  "Major mode for showing occur matches of VLF opened files."
+  (add-hook 'write-file-functions 'vlf-occur-save nil t))
 
 (defun vlf-occur-next-match ()
   "Move cursor to next match."
   (interactive)
-  (if (eq (get-char-property (point) 'face) 'match)
+  (if (eq (get-text-property (point) 'face) 'match)
       (goto-char (next-single-property-change (point) 'face)))
   (goto-char (or (text-property-any (point) (point-max) 'face 'match)
                  (text-property-any (point-min) (point)
@@ -55,9 +75,9 @@
 (defun vlf-occur-prev-match ()
   "Move cursor to previous match."
   (interactive)
-  (if (eq (get-char-property (point) 'face) 'match)
+  (if (eq (get-text-property (point) 'face) 'match)
       (goto-char (previous-single-property-change (point) 'face)))
-  (while (not (eq (get-char-property (point) 'face) 'match))
+  (while (not (eq (get-text-property (point) 'face) 'match))
     (goto-char (or (previous-single-property-change (point) 'face)
                    (point-max)))))
 
@@ -90,161 +110,370 @@ EVENT may hold details of the invocation."
     (set-buffer (window-buffer (posn-window (event-end event))))
     (goto-char (posn-point (event-end event))))
   (let* ((pos (point))
-         (pos-relative (- pos (line-beginning-position) 1))
+         (pos-relative (- pos (previous-single-char-property-change
-         (file (get-char-property pos 'file)))
+                               pos 'vlf-match)))
-    (if file
+         (chunk-start (get-text-property pos 'chunk-start)))
-        (let ((chunk-start (get-char-property pos 'chunk-start))
+    (if chunk-start
-              (chunk-end (get-char-property pos 'chunk-end))
+        (let ((chunk-end (get-text-property pos 'chunk-end))
-              (vlf-buffer (get-char-property pos 'buffer))
+              (file (if (file-exists-p vlf-occur-vlf-file)
+                        vlf-occur-vlf-file
+                      (setq vlf-occur-vlf-file
+                            (read-file-name
+                             (concat vlf-occur-vlf-file
+                                     " doesn't exist, locate it: ")))))
+              (vlf-buffer vlf-occur-vlf-buffer)
+              (not-hexl (not vlf-occur-hexl))
               (occur-buffer (current-buffer))
-              (match-pos (+ (get-char-property pos 'line-pos)
+              (match-pos (+ (get-text-property pos 'line-pos)
                             pos-relative)))
           (cond (current-prefix-arg
-                 (setq vlf-buffer (vlf file))
+                 (let ((original-occur-buffer vlf-occur-vlf-buffer))
+                   (setq vlf-buffer (vlf file t))
+                   (if (buffer-live-p original-occur-buffer)
+                       (vlf-tune-copy-profile original-occur-buffer)))
+                 (or not-hexl (hexl-mode))
                  (switch-to-buffer occur-buffer))
                 ((not (buffer-live-p vlf-buffer))
-                 (or (catch 'found
+                 (unless (catch 'found
-                       (dolist (buf (buffer-list))
+                           (dolist (buf (buffer-list))
-                         (set-buffer buf)
+                             (set-buffer buf)
-                         (and vlf-mode (equal file buffer-file-name)
+                             (and vlf-mode
-                              (setq vlf-buffer buf)
+                                  (equal file buffer-file-name)
-                              (throw 'found t))))
+                                  (eq (not (derived-mode-p 'hexl-mode))
-                     (setq vlf-buffer (vlf file)))
+                                      not-hexl)
-                 (switch-to-buffer occur-buffer)))
+                                  (setq vlf-buffer buf)
+                                  (throw 'found t))))
+                   (setq vlf-buffer (vlf file t))
+                   (or not-hexl (hexl-mode)))
+                 (switch-to-buffer occur-buffer)
+                 (setq vlf-occur-vlf-buffer vlf-buffer)))
           (pop-to-buffer vlf-buffer)
           (vlf-move-to-chunk chunk-start chunk-end)
           (goto-char match-pos)))))
 
+(defun vlf-occur-other-buffer (regexp)
+  "Make whole file occur style index for REGEXP branching to new buffer.
+Prematurely ending indexing will still show what's found so far."
+  (let ((vlf-buffer (current-buffer))
+        (file buffer-file-name)
+        (file-size vlf-file-size)
+        (batch-size vlf-batch-size)
+        (is-hexl (derived-mode-p 'hexl-mode)))
+    (with-temp-buffer
+      (setq buffer-file-name file
+            buffer-file-truename file
+            buffer-undo-list t
+            vlf-file-size file-size)
+      (set-buffer-modified-p nil)
+      (set (make-local-variable 'vlf-batch-size) batch-size)
+      (when vlf-tune-enabled
+        (vlf-tune-copy-profile vlf-buffer)
+        (vlf-tune-batch (if is-hexl
+                            '(:hexl :raw)
+                          '(:insert :encode)) t))
+      (vlf-mode 1)
+      (if is-hexl (hexl-mode))
+      (goto-char (point-min))
+      (vlf-build-occur regexp vlf-buffer)
+      (if vlf-tune-enabled
+          (vlf-tune-copy-profile (current-buffer) vlf-buffer)))))
+
 (defun vlf-occur (regexp)
   "Make whole file occur style index for REGEXP.
 Prematurely ending indexing will still show what's found so far."
   (interactive (list (read-regexp "List lines matching regexp"
                                   (if regexp-history
                                       (car regexp-history)))))
-  (if (buffer-modified-p) ;use temporary buffer not to interfere with modifications
+  (run-hook-with-args 'vlf-before-batch-functions 'occur)
-      (let ((vlf-buffer (current-buffer))
+  (if (or (buffer-modified-p)
-            (file buffer-file-name)
+          (consp buffer-undo-list)
-            (batch-size vlf-batch-size))
+          (< vlf-batch-size vlf-start-pos))
-        (with-temp-buffer
+      (vlf-occur-other-buffer regexp)
-          (setq buffer-file-name file)
-          (set-buffer-modified-p nil)
-          (set (make-local-variable 'vlf-batch-size) batch-size)
-          (vlf-mode 1)
-          (goto-char (point-min))
-          (vlf-with-undo-disabled
-           (vlf-build-occur regexp vlf-buffer))))
     (let ((start-pos vlf-start-pos)
           (end-pos vlf-end-pos)
-          (pos (point)))
+          (pos (point))
-      (vlf-with-undo-disabled
+          (batch-size vlf-batch-size))
-       (vlf-beginning-of-file)
+      (vlf-tune-batch (if (derived-mode-p 'hexl-mode)
-       (goto-char (point-min))
+                          '(:hexl :raw)
-       (unwind-protect (vlf-build-occur regexp (current-buffer))
+                        '(:insert :encode)) t)
-         (vlf-move-to-chunk start-pos end-pos)
+      (vlf-move-to-batch 0)
-         (goto-char pos))))))
+      (goto-char (point-min))
+      (unwind-protect (vlf-build-occur regexp (current-buffer))
+        (vlf-move-to-chunk start-pos end-pos)
+        (goto-char pos)
+        (setq vlf-batch-size batch-size))))
+  (run-hook-with-args 'vlf-after-batch-functions 'occur))
 
 (defun vlf-build-occur (regexp vlf-buffer)
   "Build occur style index for REGEXP over VLF-BUFFER."
-  (let ((tramp-verbose (min 2 tramp-verbose))
+  (let* ((tramp-verbose (if (boundp 'tramp-verbose)
-        (case-fold-search t)
+                            (min tramp-verbose 1)))
-        (line 1)
+         (case-fold-search t)
-        (last-match-line 0)
+         (line 1)
-        (last-line-pos (point-min))
+         (last-match-line 0)
-        (file buffer-file-name)
+         (total-matches 0)
-        (total-matches 0)
+         (first-line-offset 0)
-        (match-end-pos (+ vlf-start-pos (position-bytes (point))))
+         (first-line-incomplete nil)
-        (occur-buffer (generate-new-buffer
+         (match-start-point (point-min))
-                       (concat "*VLF-occur " (file-name-nondirectory
+         (match-end-point match-start-point)
-                                              buffer-file-name)
+         (last-match-insert-point nil)
-                               "*")))
+         (occur-buffer (generate-new-buffer
-        (line-regexp (concat "\\(?5:[\n\C-m]\\)\\|\\(?10:"
+                        (concat "*VLF-occur " (file-name-nondirectory
-                             regexp "\\)"))
+                                               buffer-file-name)
-        (batch-step (/ vlf-batch-size 8))
+                                "*")))
-        (end-of-file nil)
+         (is-hexl (derived-mode-p 'hexl-mode))
-        (reporter (make-progress-reporter
+         (end-of-file nil)
-                   (concat "Building index for " regexp "...")
+         (time (float-time))
-                   vlf-start-pos vlf-file-size)))
+         (tune-types (if is-hexl '(:hexl :raw)
+                       '(:insert :encode)))
+         (reporter (make-progress-reporter
+                    (concat "Building index for " regexp "...")
+                    vlf-start-pos vlf-file-size)))
+    (with-current-buffer occur-buffer
+      (setq buffer-undo-list t))
     (unwind-protect
         (progn
           (while (not end-of-file)
-            (if (re-search-forward line-regexp nil t)
+            (if (re-search-forward regexp nil t)
                 (progn
-                  (setq match-end-pos (+ vlf-start-pos
+                  (setq line (+ line -1
-                                         (position-bytes
+                                (count-lines match-start-point
-                                          (match-end 0))))
+                                             (1+ (match-beginning 0))))
-                  (if (match-string 5)
+                        match-start-point (match-beginning 0)
-                      (setq line (1+ line) ; line detected
+                        match-end-point (match-end 0))
-                            last-line-pos (point))
+                  (let* ((chunk-start vlf-start-pos)
-                    (let* ((chunk-start vlf-start-pos)
+                         (chunk-end vlf-end-pos)
-                           (chunk-end vlf-end-pos)
+                         (line-pos (save-excursion
-                           (line-pos (line-beginning-position))
+                                     (goto-char match-start-point)
-                           (line-text (buffer-substring
+                                     (line-beginning-position)))
-                                       line-pos (line-end-position))))
+                         (line-text (buffer-substring
-                      (with-current-buffer occur-buffer
+                                     line-pos (line-end-position))))
-                        (unless (= line last-match-line) ;new match line
+                    (if (/= line-pos (point-min))
-                          (insert "\n:") ; insert line number
+                        (setq first-line-offset 0
-                          (let* ((overlay-pos (1- (point)))
+                              first-line-incomplete nil))
-                                 (overlay (make-overlay
+                    (with-current-buffer occur-buffer
-                                           overlay-pos
+                      (unless (= line last-match-line) ;new match line
-                                           (1+ overlay-pos))))
+                        (insert "\n:") ; insert line number
-                            (overlay-put overlay 'before-string
+                        (let* ((column-point (1- (point)))
-                                         (propertize
+                               (overlay-pos column-point)
-                                          (number-to-string line)
+                               (overlay (make-overlay
-                                          'face 'shadow)))
+                                         overlay-pos
-                          (insert (propertize line-text ; insert line
+                                         (1+ overlay-pos))))
-                                              'file file
+                          (overlay-put overlay 'before-string
-                                              'buffer vlf-buffer
+                                       (propertize
-                                              'chunk-start chunk-start
+                                        (number-to-string line)
-                                              'chunk-end chunk-end
+                                        'face 'shadow))
-                                              'mouse-face '(highlight)
+                          (overlay-put overlay 'vlf-match t)
-                                              'line-pos line-pos
+                          (setq last-match-insert-point column-point
-                                              'help-echo
+                                first-line-offset 0)))
-                                              (format "Move to line %d"
+                      (when (or first-line-incomplete
-                                                      line))))
+                                (/= line last-match-line))
-                        (setq last-match-line line
+                        (insert (propertize
-                              total-matches (1+ total-matches))
+                                 (if first-line-incomplete
-                        (let ((line-start (1+
+                                     (substring line-text
-                                           (line-beginning-position)))
+                                                first-line-incomplete)
-                              (match-pos (match-beginning 10)))
+                                   line-text)
-                          (add-text-properties ; mark match
+                                 'chunk-start chunk-start
-                           (+ line-start match-pos (- last-line-pos))
+                                 'chunk-end chunk-end
-                           (+ line-start (match-end 10)
+                                 'mouse-face '(highlight)
-                              (- last-line-pos))
+                                 'line-pos line-pos
-                           (list 'face 'match
                                  'help-echo
-                                 (format "Move to match %d"
+                                 (format "Move to line %d"
-                                         total-matches))))))))
+                                         line)))
+                        (setq first-line-incomplete nil))
+                      (setq last-match-line line
+                            total-matches (1+ total-matches))
+                      (let ((line-start (+ last-match-insert-point
+                                           first-line-offset 1
+                                           (- line-pos))))
+                        (add-text-properties ; mark match
+                         (+ line-start match-start-point)
+                         (+ line-start match-end-point)
+                         (list 'face 'match
+                               'help-echo (format "Move to match %d"
+                                                  total-matches)))))))
               (setq end-of-file (= vlf-end-pos vlf-file-size))
               (unless end-of-file
-                (let ((batch-move (- vlf-end-pos batch-step)))
+                (let ((start
-                  (vlf-move-to-batch (if (< batch-move match-end-pos)
+                       (if is-hexl
-                                         match-end-pos
+                           (progn
-                                       batch-move) t))
+                             (goto-char (point-max))
-                (goto-char (if (< vlf-start-pos match-end-pos)
+                             (forward-line -10)
-                               (or (byte-to-position (- match-end-pos
+                             (setq line
-                                                        vlf-start-pos))
+                                   (+ line
-                                   (point-min))
+                                      (if (< match-end-point (point))
-                             (point-min)))
+                                          (count-lines match-start-point
-                (setq last-match-line 0
+                                                       (point))
-                      last-line-pos (line-beginning-position))
+                                        (goto-char match-end-point)
-                (progress-reporter-update reporter vlf-end-pos))))
+                                        (1- (count-lines match-start-point
+                                                         match-end-point)))))
+                             (- vlf-end-pos (* (- 10 (forward-line 10))
+                                               hexl-bits)))
+                         (let* ((pmax (point-max))
+                                (batch-step (min 1024 (/ vlf-batch-size
+                                                         10)))
+                                (batch-point
+                                 (max match-end-point
+                                      (or
+                                       (byte-to-position
+                                        (- vlf-batch-size batch-step))
+                                       (progn
+                                         (goto-char pmax)
+                                         (let ((last (line-beginning-position)))
+                                           (if (= last (point-min))
+                                               (1- (point))
+                                             last)))))))
+                           (goto-char batch-point)
+                           (setq first-line-offset
+                                 (- batch-point (line-beginning-position))
+                                 line
+                                 (+ line
+                                    (count-lines match-start-point
+                                                 batch-point)
+                                    (if (< 0 first-line-offset) -1 0)))
+                           ;; last match is on the last line?
+                           (goto-char match-end-point)
+                           (forward-line)
+                           (setq first-line-incomplete
+                                 (if (= (point) pmax)
+                                     (- pmax match-end-point)))
+                           (vlf-byte-position batch-point)))))
+                  (vlf-tune-batch tune-types)
+                  (setq vlf-end-pos start) ;not to adjust start
+                  (vlf-move-to-chunk start (+ start vlf-batch-size)))
+                (setq match-start-point (point-min)
+                      match-end-point match-start-point)
+                (goto-char match-end-point)
+                (progress-reporter-update reporter vlf-start-pos))))
           (progress-reporter-done reporter))
       (set-buffer-modified-p nil)
       (if (zerop total-matches)
-          (progn (with-current-buffer occur-buffer
+          (progn (kill-buffer occur-buffer)
-                   (set-buffer-modified-p nil))
+                 (message "No matches for \"%s\" (%f secs)"
-                 (kill-buffer occur-buffer)
+                          regexp (- (float-time) time)))
-                 (message "No matches for \"%s\"" regexp))
+        (let ((file buffer-file-name)
-        (with-current-buffer occur-buffer
+              (dir default-directory))
-          (goto-char (point-min))
+          (with-current-buffer occur-buffer
-          (insert (propertize
+            (insert "\n")
-                   (format "%d matches in %d lines for \"%s\" \
+            (goto-char (point-min))
+            (insert (propertize
+                     (format "%d matches from %d lines for \"%s\" \
 in file: %s" total-matches line regexp file)
-                   'face 'underline))
+                     'face 'underline))
-          (set-buffer-modified-p nil)
+            (set-buffer-modified-p nil)
-          (forward-char 2)
+            (forward-char 2)
-          (vlf-occur-mode))
+            (vlf-occur-mode)
-        (display-buffer occur-buffer)))))
+            (setq default-directory dir
+                  vlf-occur-vlf-file file
+                  vlf-occur-vlf-buffer vlf-buffer
+                  vlf-occur-regexp regexp
+                  vlf-occur-hexl is-hexl
+                  vlf-occur-lines line)))
+        (display-buffer occur-buffer)
+        (message "Occur finished for \"%s\" (%f secs)"
+                 regexp (- (float-time) time))))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; save, load vlf-occur data
+
+(defun vlf-occur-save (file)
+  "Serialize `vlf-occur' results to FILE which can later be reloaded."
+  (interactive (list (or buffer-file-name
+                         (read-file-name "Save vlf-occur results in: "
+                                         nil nil nil
+                                         (concat
+                                          (file-name-nondirectory
+                                           vlf-occur-vlf-file)
+                                          ".vlfo")))))
+  (setq buffer-file-name file)
+  (let ((vlf-occur-save-buffer
+         (generate-new-buffer (concat "*VLF-occur-save "
+                                      (file-name-nondirectory file)
+                                      "*"))))
+    (with-current-buffer vlf-occur-save-buffer
+      (setq buffer-file-name file
+            buffer-undo-list t)
+      (insert ";; -*- eval: (vlf-occur-load) -*-\n"))
+    (prin1 (list vlf-occur-vlf-file vlf-occur-regexp vlf-occur-hexl
+                 vlf-occur-lines)
+           vlf-occur-save-buffer)
+    (save-excursion
+      (goto-char (point-min))
+      (let ((pmax (point-max)))
+        (while (/= pmax (goto-char (next-single-char-property-change
+                                    (1+ (point)) 'vlf-match)))
+          (let* ((pos (1+ (point)))
+                 (line (get-char-property (1- pos) 'before-string)))
+            (if line
+                (prin1 (list (string-to-number line)
+                             (get-text-property pos 'chunk-start)
+                             (get-text-property pos 'chunk-end)
+                             (get-text-property pos 'line-pos)
+                             (buffer-substring-no-properties
+                              pos (1- (next-single-char-property-change
+                                       pos 'vlf-match))))
+                       vlf-occur-save-buffer))))))
+    (with-current-buffer vlf-occur-save-buffer
+      (save-buffer))
+    (kill-buffer vlf-occur-save-buffer))
+  t)
+
+;;;###autoload
+(defun vlf-occur-load ()
+  "Load serialized `vlf-occur' results from current buffer."
+  (interactive)
+  (goto-char (point-min))
+  (let* ((vlf-occur-data-buffer (current-buffer))
+         (header (read vlf-occur-data-buffer))
+         (vlf-file (nth 0 header))
+         (regexp (nth 1 header))
+         (all-lines (nth 3 header))
+         (file buffer-file-name)
+         (vlf-occur-buffer
+          (generate-new-buffer (concat "*VLF-occur "
+                                       (file-name-nondirectory file)
+                                       "*"))))
+    (switch-to-buffer vlf-occur-buffer)
+    (setq buffer-file-name file
+          buffer-undo-list t)
+    (goto-char (point-min))
+    (let ((match-count 0)
+          (form 0))
+      (while (setq form (ignore-errors (read vlf-occur-data-buffer)))
+        (goto-char (point-max))
+        (insert "\n:")
+        (let* ((overlay-pos (1- (point)))
+               (overlay (make-overlay overlay-pos (1+ overlay-pos)))
+               (line (number-to-string (nth 0 form)))
+               (pos (point)))
+          (overlay-put overlay 'before-string
+                       (propertize line 'face 'shadow))
+          (overlay-put overlay 'vlf-match t)
+          (insert (propertize (nth 4 form) 'chunk-start (nth 1 form)
+                              'chunk-end (nth 2 form)
+                              'mouse-face '(highlight)
+                              'line-pos (nth 3 form)
+                              'help-echo (concat "Move to line "
+                                                 line)))
+          (goto-char pos)
+          (while (re-search-forward regexp nil t)
+            (add-text-properties
+             (match-beginning 0) (match-end 0)
+             (list 'face 'match 'help-echo
+                   (format "Move to match %d"
+                           (setq match-count (1+ match-count))))))))
+      (kill-buffer vlf-occur-data-buffer)
+      (goto-char (point-min))
+      (insert (propertize
+               (format "%d matches from %d lines for \"%s\" in file: %s"
+                       match-count all-lines regexp vlf-file)
+               'face 'underline)))
+    (set-buffer-modified-p nil)
+    (vlf-occur-mode)
+    (setq vlf-occur-vlf-file vlf-file
+          vlf-occur-regexp regexp
+          vlf-occur-hexl (nth 2 header)
+          vlf-occur-lines all-lines)))
 
 (provide 'vlf-occur)
 

vlf-search.el

@@ -29,25 +29,38 @@
 
 (require 'vlf)
 
-(defun vlf-re-search (regexp count backward batch-step)
+(defvar hexl-bits)
+(defvar tramp-verbose)
+
+(defun vlf-re-search (regexp count backward
+                             &optional reporter time highlight)
   "Search for REGEXP COUNT number of times forward or BACKWARD.
-BATCH-STEP is amount of overlap between successive chunks."
+Use existing REPORTER and start TIME if given.
+Highlight match if HIGHLIGHT is non nil.
+Return t if search has been at least partially successful."
   (if (<= count 0)
       (error "Count must be positive"))
-  (let* ((tramp-verbose (min 2 tramp-verbose))
+  (run-hook-with-args 'vlf-before-batch-functions 'search)
+  (or reporter (setq reporter (make-progress-reporter
+                               (concat "Searching for " regexp "...")
+                               (if backward
+                                   (- vlf-file-size vlf-end-pos)
+                                 vlf-start-pos)
+                               vlf-file-size)))
+  (or time (setq time (float-time)))
+  (let* ((tramp-verbose (if (boundp 'tramp-verbose)
+                            (min tramp-verbose 1)))
          (case-fold-search t)
          (match-chunk-start vlf-start-pos)
          (match-chunk-end vlf-end-pos)
-         (match-start-pos (+ vlf-start-pos (position-bytes (point))))
+         (match-start-pos (point))
          (match-end-pos match-start-pos)
+         (last-match-pos match-start-pos)
          (to-find count)
-         (font-lock font-lock-mode)
+         (is-hexl (derived-mode-p 'hexl-mode))
-         (reporter (make-progress-reporter
+         (tune-types (if is-hexl '(:hexl :raw)
-                    (concat "Searching for " regexp "...")
+                       '(:insert :encode)))
-                    (if backward
+         (font-lock font-lock-mode))
-                        (- vlf-file-size vlf-end-pos)
-                      vlf-start-pos)
-                    vlf-file-size)))
     (font-lock-mode 0)
     (vlf-with-undo-disabled
      (unwind-protect
@@ -58,28 +71,31 @@ BATCH-STEP is amount of overlap between successive chunks."
                         (setq to-find (1- to-find)
                               match-chunk-start vlf-start-pos
                               match-chunk-end vlf-end-pos
-                              match-start-pos (+ vlf-start-pos
+                              match-start-pos (match-beginning 0)
-                                                 (position-bytes
+                              match-end-pos (match-end 0)
-                                                  (match-beginning 0)))
+                              last-match-pos match-start-pos))
-                              match-end-pos (+ vlf-start-pos
-                                               (position-bytes
-                                                (match-end 0)))))
                        ((zerop vlf-start-pos)
                         (throw 'end-of-file nil))
-                       (t (let ((batch-move (- vlf-start-pos
+                       (t (let ((end
-                                               (- vlf-batch-size
+                                 (if is-hexl
-                                                  batch-step))))
+                                     (progn
-                            (vlf-move-to-batch
+                                       (goto-char (point-min))
-                             (if (< match-start-pos batch-move)
+                                       (forward-line 10)
-                                 (- match-start-pos vlf-batch-size)
+                                       (if (< last-match-pos (point))
-                               batch-move) t))
+                                           (goto-char last-match-pos))
-                          (goto-char (if (< match-start-pos
+                                       (+ vlf-start-pos
-                                            vlf-end-pos)
+                                          (* (- 10 (forward-line -10))
-                                         (or (byte-to-position
+                                             hexl-bits)))
-                                              (- match-start-pos
+                                   (vlf-byte-position
-                                                 vlf-start-pos))
+                                    (min 1024 (/ (point-max) 10)
-                                             (point-max))
+                                         last-match-pos)))))
-                                       (point-max)))
+                            (vlf-tune-batch tune-types)
+                            (setq vlf-start-pos end) ;don't adjust end
+                            (vlf-move-to-chunk (- end vlf-batch-size)
+                                               end))
+                          (let ((pmax (point-max)))
+                            (goto-char pmax)
+                            (setq last-match-pos pmax))
                           (progress-reporter-update
                            reporter (- vlf-file-size
                                        vlf-start-pos)))))
@@ -88,72 +104,80 @@ BATCH-STEP is amount of overlap between successive chunks."
                       (setq to-find (1- to-find)
                             match-chunk-start vlf-start-pos
                             match-chunk-end vlf-end-pos
-                            match-start-pos (+ vlf-start-pos
+                            match-start-pos (match-beginning 0)
-                                               (position-bytes
+                            match-end-pos (match-end 0)
-                                                (match-beginning 0)))
+                            last-match-pos match-end-pos))
-                            match-end-pos (+ vlf-start-pos
+                     ((>= vlf-end-pos vlf-file-size)
-                                             (position-bytes
-                                              (match-end 0)))))
-                     ((= vlf-end-pos vlf-file-size)
                       (throw 'end-of-file nil))
-                     (t (let ((batch-move (- vlf-end-pos batch-step)))
+                     (t (let* ((pmax (point-max))
-                          (vlf-move-to-batch
+                               (start
-                           (if (< batch-move match-end-pos)
+                                (if is-hexl
-                               match-end-pos
+                                    (progn
-                             batch-move) t))
+                                      (goto-char pmax)
-                        (goto-char (if (< vlf-start-pos match-end-pos)
+                                      (forward-line -10)
-                                       (or (byte-to-position
+                                      (if (< (point) last-match-pos)
-                                            (- match-end-pos
+                                          (goto-char last-match-pos))
-                                               vlf-start-pos))
+                                      (- vlf-end-pos
-                                           (point-min))
+                                         (* (- 10 (forward-line 10))
-                                     (point-min)))
+                                            hexl-bits)))
+                                  (vlf-byte-position
+                                   (max (- pmax 1024)
+                                        (- pmax (/ pmax 10))
+                                        last-match-pos)))))
+                          (vlf-tune-batch tune-types)
+                          (setq vlf-end-pos start) ;don't adjust start
+                          (vlf-move-to-chunk start (+ start
+                                                      vlf-batch-size)))
+                        (let ((pmin (point-min)))
+                          (goto-char pmin)
+                          (setq last-match-pos pmin))
                         (progress-reporter-update reporter
                                                   vlf-end-pos)))))
            (progress-reporter-done reporter))
        (set-buffer-modified-p nil)
        (if font-lock (font-lock-mode 1))
-       (if backward
+       (let ((result
-           (vlf-goto-match match-chunk-start match-chunk-end
+              (if backward
-                           match-end-pos match-start-pos
+                  (vlf-goto-match match-chunk-start match-chunk-end
-                           count to-find)
+                                  match-end-pos match-start-pos
-         (vlf-goto-match match-chunk-start match-chunk-end
+                                  count to-find time highlight)
-                         match-start-pos match-end-pos
+                (vlf-goto-match match-chunk-start match-chunk-end
-                         count to-find))))))
+                                match-start-pos match-end-pos
+                                count to-find time highlight))))
+         (run-hook-with-args 'vlf-after-batch-functions 'search)
+         result)))))
 
 (defun vlf-goto-match (match-chunk-start match-chunk-end
-                                         match-pos-start
+                                         match-start-pos match-end-pos
-                                         match-pos-end
+                                         count to-find time
-                                         count to-find)
+                                         highlight)
-  "Move to MATCH-CHUNK-START MATCH-CHUNK-END surrounding \
+  "Move to MATCH-CHUNK-START MATCH-CHUNK-END surrounding\
-MATCH-POS-START and MATCH-POS-END.
+MATCH-START-POS and MATCH-END-POS.
 According to COUNT and left TO-FIND, show if search has been
-successful.  Return nil if nothing found."
+successful.  Use start TIME to report how much it took.
+Highlight match if HIGHLIGHT is non nil.
+Return nil if nothing found."
+  (vlf-move-to-chunk match-chunk-start match-chunk-end)
+  (goto-char match-start-pos)
+  (setq vlf-batch-size (vlf-tune-optimal-load
+                        (if (derived-mode-p 'hexl-mode)
+                            '(:hexl :raw)
+                          '(:insert :encode))))
   (if (= count to-find)
-      (progn (vlf-move-to-chunk match-chunk-start match-chunk-end)
+      (progn (message "Not found (%f secs)" (- (float-time) time))
-             (goto-char (or (byte-to-position (- match-pos-start
-                                                 vlf-start-pos))
-                            (point-max)))
-             (message "Not found")
              nil)
-    (let ((success (zerop to-find)))
+    (let ((success (zerop to-find))
+          (overlay (make-overlay match-start-pos match-end-pos)))
+      (overlay-put overlay 'face 'match)
       (if success
-          (vlf-update-buffer-name)
+          (message "Match found (%f secs)" (- (float-time) time))
-        (vlf-move-to-chunk match-chunk-start match-chunk-end))
+        (message "Moved to the %d match which is last (%f secs)"
-      (let* ((match-end (or (byte-to-position (- match-pos-end
+                 (- count to-find) (- (float-time) time)))
-                                                 vlf-start-pos))
+      (if highlight
-                            (point-max)))
+          (unwind-protect (sit-for 1)
-             (overlay (make-overlay (byte-to-position
+            (delete-overlay overlay))
-                                     (- match-pos-start
+        (delete-overlay overlay)))
-                                        vlf-start-pos))
+    t))
-                                    match-end)))
-        (overlay-put overlay 'face 'match)
-        (unless success
-          (goto-char match-end)
-          (message "Moved to the %d match which is last"
-                   (- count to-find)))
-        (unwind-protect (sit-for 3)
-          (delete-overlay overlay))
-        t))))
 
 (defun vlf-re-search-forward (regexp count)
   "Search forward for REGEXP prefix COUNT number of times.
@@ -163,7 +187,11 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (vlf-re-search regexp count nil (/ vlf-batch-size 8)))
+  (let ((batch-size vlf-batch-size)
+        success)
+    (unwind-protect
+        (setq success (vlf-re-search regexp count nil nil nil t))
+      (or success (setq vlf-batch-size batch-size)))))
 
 (defun vlf-re-search-backward (regexp count)
   "Search backward for REGEXP prefix COUNT number of times.
@@ -173,78 +201,152 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (vlf-re-search regexp count t (/ vlf-batch-size 8)))
+  (let ((batch-size vlf-batch-size)
+        success)
+    (unwind-protect
+        (setq success (vlf-re-search regexp count t nil nil t))
+      (or success (setq vlf-batch-size batch-size)))))
 
 (defun vlf-goto-line (n)
   "Go to line N.  If N is negative, count from the end of file."
   (interactive (if (vlf-no-modifications)
                    (list (read-number "Go to line: "))))
-  (vlf-verify-size)
+  (if (derived-mode-p 'hexl-mode)
-  (let ((tramp-verbose (min 2 tramp-verbose))
+      (vlf-goto-line-hexl n)
-        (start-pos vlf-start-pos)
+    (run-hook-with-args 'vlf-before-batch-functions 'goto-line)
-        (end-pos vlf-end-pos)
+    (vlf-verify-size)
-        (pos (point))
+    (let ((tramp-verbose (if (boundp 'tramp-verbose)
-        (font-lock font-lock-mode)
+                             (min tramp-verbose 1)))
-        (success nil))
+          (start-pos vlf-start-pos)
-    (font-lock-mode 0)
+          (end-pos vlf-end-pos)
-    (unwind-protect
+          (batch-size vlf-batch-size)
-        (if (< 0 n)
+          (pos (point))
-            (let ((start 0)
+          (font-lock font-lock-mode)
-                  (end (min vlf-batch-size vlf-file-size))
+          (time (float-time))
+          (success nil))
+      (font-lock-mode 0)
+      (vlf-tune-batch '(:raw))
+      (unwind-protect
+          (if (< 0 n)
+              (let ((start 0)
+                    (end (min vlf-batch-size vlf-file-size))
+                    (reporter (make-progress-reporter
+                               (concat "Searching for line "
+                                       (number-to-string n) "...")
+                               0 vlf-file-size))
+                    (inhibit-read-only t))
+                (setq n (1- n))
+                (vlf-with-undo-disabled
+                 ;; (while (and (< (- end start) n)
+                 ;;             (< n (- vlf-file-size start)))
+                 ;;   (erase-buffer)
+                 ;;   (vlf-tune-insert-file-contents-literally start end)
+                 ;;   (goto-char (point-min))
+                 ;;   (while (re-search-forward "[\n\C-m]" nil t)
+                 ;;     (setq n (1- n)))
+                 ;;   (vlf-verify-size)
+                 ;;   (vlf-tune-batch '(:raw))
+                 ;;   (setq start end
+                 ;;         end (min vlf-file-size (+ start
+                 ;;                                   vlf-batch-size)))
+                 ;;   (progress-reporter-update reporter start))
+                 (when (< n (- vlf-file-size end))
+                   (vlf-tune-batch '(:insert :encode))
+                   (vlf-move-to-chunk start (+ start vlf-batch-size))
+                   (goto-char (point-min))
+                   (setq success
+                         (or (zerop n)
+                             (when (vlf-re-search "[\n\C-m]" n nil
+                                                  reporter time)
+                               (forward-char) t))))))
+            (let ((end vlf-file-size)
                   (reporter (make-progress-reporter
-                             (concat "Searching for line "
+                             (concat "Searching for line -"
                                      (number-to-string n) "...")
                              0 vlf-file-size))
                   (inhibit-read-only t))
-              (setq n (1- n))
+              (setq n (- n))
               (vlf-with-undo-disabled
-               (while (and (< (- end start) n)
+               ;; (let ((start (max 0 (- vlf-file-size vlf-batch-size))))
-                           (< n (- vlf-file-size start)))
+               ;;   (while (and (< (- end start) n) (< n end))
-                 (erase-buffer)
+               ;;     (erase-buffer)
-                 (insert-file-contents-literally buffer-file-name
+               ;;     (vlf-tune-insert-file-contents-literally start end)
-                                                 nil start end)
+               ;;     (goto-char (point-max))
-                 (goto-char (point-min))
+               ;;     (while (re-search-backward "[\n\C-m]" nil t)
-                 (while (re-search-forward "[\n\C-m]" nil t)
+               ;;       (setq n (1- n)))
-                   (setq n (1- n)))
+               ;;     (vlf-tune-batch '(:raw))
-                 (vlf-verify-size)
+               ;;     (setq end start
-                 (setq start end
+               ;;           start (max 0 (- end vlf-batch-size)))
-                       end (min vlf-file-size
+               ;;     (progress-reporter-update reporter
-                                (+ start vlf-batch-size)))
+               ;;                               (- vlf-file-size end))))
-                 (progress-reporter-update reporter start))
+               (when (< n end)
-               (when (< n (- vlf-file-size end))
+                 (vlf-tune-batch '(:insert :encode))
-                 (vlf-move-to-chunk-2 start end)
+                 (vlf-move-to-chunk (- end vlf-batch-size) end)
-                 (goto-char (point-min))
+                 (goto-char (point-max))
-                 (setq success (vlf-re-search "[\n\C-m]" n nil 0)))))
+                 (setq success (vlf-re-search "[\n\C-m]" n t
-          (let ((start (max 0 (- vlf-file-size vlf-batch-size)))
+                                              reporter time))))))
-                (end vlf-file-size)
+        (if font-lock (font-lock-mode 1))
-                (reporter (make-progress-reporter
+        (unless success
-                           (concat "Searching for line -"
+          (vlf-with-undo-disabled
-                                   (number-to-string n) "...")
+           (vlf-move-to-chunk start-pos end-pos))
-                           0 vlf-file-size))
+          (goto-char pos)
-                (inhibit-read-only t))
+          (setq vlf-batch-size batch-size)
-            (setq n (- n))
+          (message "Unable to find line"))
-            (vlf-with-undo-disabled
+        (run-hook-with-args 'vlf-after-batch-functions 'goto-line)))))
-             (while (and (< (- end start) n) (< n end))
+
-               (erase-buffer)
+(defun vlf-goto-line-hexl (n)
-               (insert-file-contents-literally buffer-file-name nil
+  "Go to line N.  If N is negative, count from the end of file.
-                                               start end)
+Assume `hexl-mode' is active."
-               (goto-char (point-max))
+  (vlf-tune-load '(:hexl :raw))
-               (while (re-search-backward "[\n\C-m]" nil t)
+  (if (< n 0)
-                 (setq n (1- n)))
+      (let ((hidden-bytes (+ vlf-file-size (* n hexl-bits))))
-               (setq end start
+        (setq hidden-bytes (- hidden-bytes (mod hidden-bytes
-                     start (max 0 (- end vlf-batch-size)))
+                                                vlf-batch-size)))
-               (progress-reporter-update reporter
+        (vlf-move-to-batch hidden-bytes)
-                                         (- vlf-file-size end)))
+        (goto-char (point-max))
-             (when (< n end)
+        (forward-line (+ (round (- vlf-file-size
-               (vlf-move-to-chunk-2 start end)
+                                   (min vlf-file-size
-               (goto-char (point-max))
+                                        (+ hidden-bytes
-               (setq success (vlf-re-search "[\n\C-m]" n t 0))))))
+                                           vlf-batch-size)))
-      (if font-lock (font-lock-mode 1))
+                                hexl-bits)
-      (unless success
+                         n)))
-        (vlf-with-undo-disabled
+    (let ((hidden-bytes (1- (* n hexl-bits))))
-         (vlf-move-to-chunk-2 start-pos end-pos))
+      (setq hidden-bytes (- hidden-bytes (mod hidden-bytes
-        (goto-char pos)
+                                              vlf-batch-size)))
-        (message "Unable to find line")))))
+      (vlf-move-to-batch hidden-bytes)
+      (goto-char (point-min))
+      (forward-line (- n 1 (/ hidden-bytes hexl-bits))))))
+
+(defun vlf-query-replace (regexp to-string &optional delimited backward)
+  "Query replace over whole file matching REGEXP with TO-STRING.
+Third arg DELIMITED (prefix arg if interactive), if non-nil, replace
+only matches surrounded by word boundaries.  A negative prefix arg means
+replace BACKWARD."
+  (interactive (let ((common (query-replace-read-args
+                              (concat "Query replace over whole file"
+                                      (if current-prefix-arg
+                                          (if (eq current-prefix-arg '-)
+                                              " backward"
+                                            " word")
+                                        "")
+                                      " regexp")
+                              t)))
+                 (list (nth 0 common) (nth 1 common) (nth 2 common)
+                       (nth 3 common))))
+  (let ((not-automatic t))
+    (while (vlf-re-search regexp 1 backward)
+      (cond (not-automatic
+             (query-replace-regexp regexp to-string delimited
+                                   nil nil backward)
+             (if (eq 'automatic (lookup-key query-replace-map
+                                            (vector last-input-event)))
+                 (setq not-automatic nil)))
+            (backward (while (re-search-backward regexp nil t)
+                        (replace-match to-string)))
+            (t (while (re-search-forward regexp nil t)
+                 (replace-match to-string))))
+      (if (buffer-modified-p)
+          (save-buffer)))))
 
 (provide 'vlf-search)
 

vlf-setup.el

@@ -1,6 +1,6 @@
-;;; vlf-integrate.el --- VLF integration with other packages  -*- lexical-binding: t -*-
+;;; vlf-setup.el --- VLF integration with other packages  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2014 Free Software Foundation, Inc.
+;; Copyright (C) 2014-2015 Free Software Foundation, Inc.
 
 ;; Keywords: large files, integration
 ;; Author: Andrey Kotlarski <m00naticus@gmail.com>
@@ -26,10 +26,12 @@
 
 ;;; Code:
 
-(defgroup vlf nil
+(defgroup vlf nil "View Large Files in Emacs."
-  "View Large Files in Emacs."
+  :prefix "vlf-" :group 'files)
-  :prefix "vlf-"
+
-  :group 'files)
+(defcustom vlf-batch-size 1000000
+  "Defines how large each batch of file data initially is (in bytes)."
+  :group 'vlf :type 'integer)
 
 (defcustom vlf-application 'ask
   "Determines when `vlf' will be offered on opening files.
@@ -37,18 +39,19 @@ Possible values are: nil to never use it;
 `ask' offer `vlf' when file size is beyond `large-file-warning-threshold';
 `dont-ask' automatically use `vlf' for large files;
 `always' use `vlf' for all files."
-  :group 'vlf
+  :group 'vlf :type '(radio (const :format "%v " nil)
-  :type '(radio (const :format "%v " nil)
+                            (const :format "%v " ask)
-                (const :format "%v " ask)
+                            (const :format "%v " dont-ask)
-                (const :format "%v " dont-ask)
+                            (const :format "%v" always)))
-                (const :format "%v" always)))
 
 (defcustom vlf-forbidden-modes-list
   '(archive-mode tar-mode jka-compr git-commit-mode image-mode
                  doc-view-mode doc-view-mode-maybe ebrowse-tree-mode)
   "Major modes which VLF will not be automatically applied to."
-  :group 'vlf
+  :group 'vlf :type '(list symbol))
-  :type '(list symbol))
+
+(defvar dired-mode-map)
+(declare-function dired-get-file-for-visit "dired")
 
 (unless (fboundp 'file-size-human-readable)
   (defun file-size-human-readable (file-size)
@@ -102,7 +105,8 @@ OP-TYPE specifies the file operation being performed over FILENAME."
     (vlf filename)
     (error ""))
    ((and large-file-warning-threshold
-         (< large-file-warning-threshold size))
+         (< large-file-warning-threshold size)
+         (< vlf-batch-size size))
     (if (eq vlf-application 'dont-ask)
         (progn (vlf filename)
                (error ""))
@@ -151,6 +155,6 @@ defined in FILE."
 (eval-after-load "dired"
   '(define-key dired-mode-map "V" 'dired-vlf))
 
-(provide 'vlf-integrate)
+(provide 'vlf-setup)
 
-;;; vlf-integrate.el ends here
+;;; vlf-setup.el ends here

vlf-tune.el

@@ -0,0 +1,462 @@
+;;; vlf-tune.el --- VLF tuning operations  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2014 Free Software Foundation, Inc.
+
+;; Keywords: large files, batch size, performance
+;; Author: Andrey Kotlarski <m00naticus@gmail.com>
+;; URL: https://github.com/m00natic/vlfi
+
+;; This file is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+
+;; This file is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs; see the file COPYING.  If not, write to
+;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+;; Boston, MA 02111-1307, USA.
+
+;;; Commentary:
+;; This package provides wrappers for basic chunk operations that add
+;; profiling and automatic tuning of `vlf-batch-size'.
+
+;;; Code:
+
+(defgroup vlf nil "View Large Files in Emacs."
+  :prefix "vlf-" :group 'files)
+
+(defcustom vlf-batch-size 1000000
+  "Defines how large each batch of file data initially is (in bytes)."
+  :group 'vlf :type 'integer)
+(put 'vlf-batch-size 'permanent-local t)
+
+(defcustom vlf-tune-enabled t
+  "Whether to allow automatic change of batch size.
+If nil, completely disable.  If `stats', maintain measure statistics,
+but don't change batch size.  If t, measure and change."
+  :group 'vlf :type '(choice (const :tag "Enabled" t)
+                             (const :tag "Just statistics" stats)
+                             (const :tag "Disabled" nil)))
+
+(defvar vlf-file-size 0 "Total size in bytes of presented file.")
+(make-variable-buffer-local 'vlf-file-size)
+(put 'vlf-file-size 'permanent-local t)
+
+(defun vlf-tune-ram-size ()
+  "Try to determine RAM size in bytes."
+  (if (executable-find "free")
+      (let* ((free (shell-command-to-string "free"))
+             (match-from (string-match "[[:digit:]]+" free)))
+        (if match-from
+            (* 1000 (string-to-number (substring free match-from
+                                                 (match-end 0))))))))
+
+(defcustom vlf-tune-max (max (let ((ram-size (vlf-tune-ram-size)))
+                               (if ram-size
+                                   (/ ram-size 20)
+                                 0))
+                             large-file-warning-threshold)
+  "Maximum batch size in bytes when auto tuning.
+Avoid increasing this after opening file with VLF."
+  :group 'vlf :type 'integer)
+
+(defcustom vlf-tune-step (/ vlf-tune-max 10000)
+  "Step used for tuning in bytes.
+Avoid decreasing this after opening file with VLF."
+  :group 'vlf :type 'integer)
+
+(defcustom vlf-tune-load-time 1.0
+  "How many seconds should batch take to load for best user experience."
+  :group 'vlf :type 'float)
+
+(defvar vlf-tune-insert-bps nil
+  "Vector of bytes per second insert measurements.")
+(make-variable-buffer-local 'vlf-tune-insert-bps)
+(put 'vlf-tune-insert-bps 'permanent-local t)
+
+(defvar vlf-tune-insert-raw-bps nil
+  "Vector of bytes per second non-decode insert measurements.")
+(make-variable-buffer-local 'vlf-tune-insert-raw-bps)
+(put 'vlf-tune-insert-raw-bps 'permanent-local t)
+
+(defvar vlf-tune-encode-bps nil
+  "Vector of bytes per second encode measurements.")
+(make-variable-buffer-local 'vlf-tune-encode-bps)
+(put 'vlf-tune-encode-bps 'permanent-local t)
+
+(defvar vlf-tune-write-bps nil
+  "Vector of bytes per second write measurements.")
+
+(defvar vlf-tune-hexl-bps nil
+  "Vector of bytes per second hexlify measurements.")
+
+(defvar vlf-tune-dehexlify-bps nil
+  "Vector of bytes per second dehexlify measurements.")
+
+(defvar vlf-start-pos)
+(defvar hexl-bits)
+(defvar hexl-max-address)
+(declare-function hexl-line-displen "hexl")
+(declare-function dehexlify-buffer "hexl")
+
+(defun vlf-tune-copy-profile (from-buffer &optional to-buffer)
+  "Copy specific profile vectors of FROM-BUFFER to TO-BUFFER.
+If TO-BUFFER is nil, copy to current buffer."
+  (let (insert-bps insert-raw-bps encode-bps)
+    (with-current-buffer from-buffer
+      (setq insert-bps vlf-tune-insert-bps
+            insert-raw-bps vlf-tune-insert-raw-bps
+            encode-bps vlf-tune-encode-bps))
+    (if to-buffer
+        (with-current-buffer to-buffer
+          (setq vlf-tune-insert-bps insert-bps
+                vlf-tune-insert-raw-bps insert-raw-bps
+                vlf-tune-encode-bps encode-bps))
+      (setq vlf-tune-insert-bps insert-bps
+            vlf-tune-insert-raw-bps insert-raw-bps
+            vlf-tune-encode-bps encode-bps))))
+
+(defun vlf-tune-closest-index (size)
+  "Get closest measurement index corresponding to SIZE."
+  (let ((step (float vlf-tune-step)))
+    (max 0 (1- (min (round size step) (round vlf-tune-max step))))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; profiling
+
+(defun vlf-tune-initialize-measurement ()
+  "Initialize measurement vector."
+  (make-vector (1- (/ vlf-tune-max vlf-tune-step)) nil))
+
+(defmacro vlf-tune-add-measurement (vec size time)
+  "Add at an appropriate position in VEC new SIZE TIME measurement.
+VEC is a vector of (mean time . count) elements ordered by size."
+  `(when (and vlf-tune-enabled (not (zerop ,size)))
+     (or ,vec (setq ,vec (vlf-tune-initialize-measurement)))
+     (let* ((idx (vlf-tune-closest-index ,size))
+            (existing (aref ,vec idx)))
+       (aset ,vec idx (if (consp existing)
+                          (let ((count (1+ (cdr existing)))) ;recalculate mean
+                            (cons (/ (+ (* (1- count) (car existing))
+                                        (/ ,size ,time))
+                                     count)
+                                  count))
+                        (cons (/ ,size ,time) 1))))))
+
+(defmacro vlf-time (&rest body)
+  "Get timing consed with result of BODY execution."
+  `(if vlf-tune-enabled
+       (let* ((time (float-time))
+              (result (progn ,@body)))
+         (cons (- (float-time) time) result))
+     (let ((result (progn ,@body)))
+       (cons nil result))))
+
+(defun vlf-tune-insert-file-contents (start end)
+  "Extract decoded file bytes START to END and save time it takes."
+  (let ((result (vlf-time (insert-file-contents buffer-file-name
+                                                nil start end))))
+    (vlf-tune-add-measurement vlf-tune-insert-bps
+                              (- end start) (car result))
+    (cdr result)))
+
+(defun vlf-tune-insert-file-contents-literally (start end &optional file)
+  "Insert raw file bytes START to END and save time it takes.
+FILE if given is filename to be used, otherwise `buffer-file-name'."
+  (let ((result (vlf-time (insert-file-contents-literally
+                           (or file buffer-file-name) nil start end))))
+    (vlf-tune-add-measurement vlf-tune-insert-raw-bps
+                              (- end start) (car result))
+    (cdr result)))
+
+(defun vlf-tune-encode-length (start end)
+  "Get length of encoded region START to END and save time it takes."
+  (let ((result (vlf-time (length (encode-coding-region
+                                   start end
+                                   buffer-file-coding-system t)))))
+    (vlf-tune-add-measurement vlf-tune-encode-bps
+                              (cdr result) (car result))
+    (cdr result)))
+
+(defun vlf-tune-write (start end append visit size &optional file-name)
+  "Save buffer and save time it takes.
+START, END, APPEND, VISIT have same meaning as in `write-region'.
+SIZE is number of bytes that are saved.
+FILE-NAME if given is to be used instead of `buffer-file-name'."
+  (let* ((file (or file-name buffer-file-name))
+         (time (car (vlf-time (write-region start end file append
+                                            visit)))))
+    (or (file-remote-p file) ;writing to remote files can include network copying
+        (vlf-tune-add-measurement vlf-tune-write-bps size time))))
+
+(defun vlf-hexl-adjust-addresses ()
+  "Adjust hexl address indicators according to `vlf-start-pos'."
+  (let ((pos (point))
+        (address vlf-start-pos))
+    (goto-char (point-min))
+    (while (re-search-forward "^[[:xdigit:]]+" nil t)
+      (replace-match (format "%08x" address))
+      (setq address (+ address hexl-bits)))
+    (goto-char pos)))
+
+(defun vlf-tune-hexlify ()
+  "Activate `hexl-mode' and save time it takes."
+  (let* ((no-adjust (zerop vlf-start-pos))
+         (time (car (vlf-time (hexlify-buffer)
+                              (or no-adjust
+                                  (vlf-hexl-adjust-addresses))))))
+    (setq hexl-max-address (+ (* (/ (1- (buffer-size))
+                                    (hexl-line-displen)) 16) 15))
+    (or no-adjust
+        (vlf-tune-add-measurement vlf-tune-hexl-bps
+                                  hexl-max-address time))))
+
+(defun vlf-tune-dehexlify ()
+  "Exit `hexl-mode' and save time it takes."
+  (let ((time (car (vlf-time (dehexlify-buffer)))))
+    (vlf-tune-add-measurement vlf-tune-dehexlify-bps
+                              hexl-max-address time)))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; tuning
+
+(defun vlf-tune-approximate-nearby (vec index)
+  "VEC has value for INDEX, approximate to closest available."
+  (let ((val 0)
+        (left-idx (1- index))
+        (right-idx (1+ index))
+        (min-idx (max 0 (- index 5)))
+        (max-idx (min (+ index 6)
+                      (1- (/ (min vlf-tune-max
+                                  (/ (1+ vlf-file-size) 2))
+                             vlf-tune-step)))))
+    (while (and (zerop val) (or (<= min-idx left-idx)
+                                (< right-idx max-idx)))
+      (if (<= min-idx left-idx)
+          (let ((left (aref vec left-idx)))
+            (cond ((consp left) (setq val (car left)))
+                  ((numberp left) (setq val left)))))
+      (if (< right-idx max-idx)
+          (let ((right (aref vec right-idx)))
+            (if (consp right)
+                (setq right (car right)))
+            (and (numberp right) (not (zerop right))
+                 (setq val (if (zerop val)
+                               right
+                             (/ (+ val right) 2))))))
+      (setq left-idx (1- left-idx)
+            right-idx (1+ right-idx)))
+    val))
+
+(defmacro vlf-tune-get-value (vec index &optional dont-approximate)
+  "Get value from VEC for INDEX.
+If missing, approximate from nearby measurement,
+unless DONT-APPROXIMATE is t."
+  `(if ,vec
+       (let ((val (aref ,vec ,index)))
+         (cond ((consp val) (car val))
+               ((null val)
+                ,(if dont-approximate
+                     `(aset ,vec ,index 0)
+                   `(vlf-tune-approximate-nearby ,vec ,index)))
+               ((zerop val) ;index has been tried before, yet still no value
+                ,(if dont-approximate
+                     `(aset ,vec ,index
+                            (vlf-tune-approximate-nearby ,vec ,index))
+                   `(vlf-tune-approximate-nearby ,vec ,index)))
+               (t val)))
+     most-positive-fixnum))
+
+(defmacro vlf-tune-get-vector (key)
+  "Get vlf-tune vector corresponding to KEY."
+  `(cond ((eq ,key :insert) vlf-tune-insert-bps)
+         ((eq ,key :raw) vlf-tune-insert-raw-bps)
+         ((eq ,key :encode) vlf-tune-encode-bps)
+         ((eq ,key :write) vlf-tune-write-bps)
+         ((eq ,key :hexl) vlf-tune-hexl-bps)
+         ((eq ,key :dehexlify) vlf-tune-dehexlify-bps)))
+
+(defun vlf-tune-assess (type coef index &optional approximate)
+  "Get measurement value according to TYPE, COEF and INDEX.
+If APPROXIMATE is t, do approximation for missing values."
+  (* coef (or (if approximate
+                  (vlf-tune-get-value (vlf-tune-get-vector type)
+                                      index)
+                (vlf-tune-get-value (vlf-tune-get-vector type)
+                                    index t))
+              0)))
+
+(defun vlf-tune-score (types index &optional approximate time-max)
+  "Calculate cumulative speed over TYPES for INDEX.
+If APPROXIMATE is t, do approximation for missing values.
+If TIME-MAX is non nil, return cumulative time instead of speed.
+If it is number, stop as soon as cumulative time gets equal or above."
+  (catch 'result
+    (let ((time 0)
+          (size (* (1+ index) vlf-tune-step))
+          (cut-time (numberp time-max)))
+      (dolist (el types (if time-max time
+                          (/ size time)))
+        (let ((bps (if (consp el)
+                       (vlf-tune-assess (car el) (cadr el) index
+                                        approximate)
+                     (vlf-tune-assess el 1.0 index approximate))))
+          (if (zerop bps)
+              (throw 'result nil)
+            (setq time (+ time (/ size bps)))
+            (and cut-time (<= time-max time)
+                 (throw 'result nil))))))))
+
+(defun vlf-tune-conservative (types &optional index)
+  "Adjust `vlf-batch-size' to best nearby value over TYPES.
+INDEX if given, specifies search independent of current batch size."
+  (if (eq vlf-tune-enabled t)
+      (let* ((half-max (/ (1+ vlf-file-size) 2))
+             (idx (or index (vlf-tune-closest-index vlf-batch-size)))
+             (curr (if (< half-max (* idx vlf-tune-step)) t
+                     (vlf-tune-score types idx))))
+        (if curr
+            (let ((prev (if (zerop idx) t
+                          (vlf-tune-score types (1- idx)))))
+              (if prev
+                  (let ((next (if (or (eq curr t)
+                                      (< half-max (* (1+ idx)
+                                                     vlf-tune-step)))
+                                  t
+                                (vlf-tune-score types (1+ idx)))))
+                    (cond ((null next)
+                           (setq vlf-batch-size (* (+ 2 idx)
+                                                   vlf-tune-step)))
+                          ((eq curr t)
+                           (or (eq prev t)
+                               (setq vlf-batch-size
+                                     (* idx vlf-tune-step))))
+                          (t (let ((best-idx idx))
+                               (and (numberp prev) (< curr prev)
+                                    (setq curr prev
+                                          best-idx (1- idx)))
+                               (and (numberp next) (< curr next)
+                                    (setq best-idx (1+ idx)))
+                               (setq vlf-batch-size
+                                     (* (1+ best-idx)
+                                        vlf-tune-step))))))
+                (setq vlf-batch-size (* idx vlf-tune-step))))
+          (setq vlf-batch-size (* (1+ idx) vlf-tune-step))))))
+
+(defun vlf-tune-binary (types min max)
+  "Adjust `vlf-batch-size' to optimal value using binary search, \
+optimizing over TYPES.
+MIN and MAX specify interval of indexes to search."
+  (let ((sum (+ min max)))
+    (if (< (- max min) 3)
+        (vlf-tune-conservative types (/ sum 2))
+      (let* ((left-idx (round (+ sum (* 2 min)) 4))
+             (left (vlf-tune-score types left-idx)))
+        (if left
+            (let* ((right-idx (round (+ sum (* 2 max)) 4))
+                   (right (vlf-tune-score types right-idx)))
+              (cond ((null right)
+                     (setq vlf-batch-size (* (1+ right-idx)
+                                             vlf-tune-step)))
+                    ((< left right)
+                     (vlf-tune-binary types (/ (1+ sum) 2) max))
+                    (t (vlf-tune-binary types min (/ sum 2)))))
+          (setq vlf-batch-size (* (1+ left-idx) vlf-tune-step)))))))
+
+(defun vlf-tune-linear (types max-idx)
+  "Adjust `vlf-batch-size' to optimal known value using linear search.
+Optimize over TYPES up to MAX-IDX."
+  (let ((best-idx 0)
+        (best-bps 0)
+        (idx 0))
+    (while (< idx max-idx)
+      (let ((bps (vlf-tune-score types idx t)))
+        (and bps (< best-bps bps)
+             (setq best-idx idx
+                   best-bps bps)))
+      (setq idx (1+ idx)))
+    (setq vlf-batch-size (* (1+ best-idx) vlf-tune-step))))
+
+(defun vlf-tune-batch (types &optional linear file)
+  "Adjust `vlf-batch-size' to optimal value optimizing on TYPES.
+TYPES is alist of elements that may be of form (type coef) or
+non list values in which case coeficient is assumed 1.
+Types can be :insert, :raw, :encode, :write, :hexl or :dehexlify.
+If LINEAR is non nil, use brute-force.  In case requested measurement
+is missing, stop search and set `vlf-batch-size' to this value.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
+Suitable for multiple batch operations."
+  (if (eq vlf-tune-enabled t)
+      (let ((max-idx (1- (/ (min vlf-tune-max
+                                 (/ (1+ vlf-file-size) 2))
+                            vlf-tune-step))))
+        (if linear
+            (vlf-tune-linear types max-idx)
+          (let ((batch-size vlf-batch-size))
+            (cond ((file-remote-p (or file buffer-file-name))
+                   (vlf-tune-conservative types))
+                  ((<= 1 max-idx)
+                   (if (< max-idx 3)
+                       (vlf-tune-conservative types (/ max-idx 2))
+                     (vlf-tune-binary types 0 max-idx))))
+            (if (= batch-size vlf-batch-size) ;local maxima?
+                (vlf-tune-linear types max-idx)))))))
+
+(defun vlf-tune-optimal-load (types &optional min-idx max-idx)
+  "Get best batch size according to existing measurements over TYPES.
+Best considered where primitive operations total is closest to
+`vlf-tune-load-time'.  If MIN-IDX and MAX-IDX are given,
+confine search to this region."
+  (if (eq vlf-tune-enabled t)
+      (progn
+        (setq min-idx (max 0 (or min-idx 0))
+              max-idx (min (or max-idx vlf-tune-max)
+                           (1- (/ (min vlf-tune-max
+                                       (/ (1+ vlf-file-size) 2))
+                                  vlf-tune-step))))
+        (let* ((idx min-idx)
+               (best-idx idx)
+               (best-time-diff vlf-tune-load-time)
+               (all-less t)
+               (all-more t))
+          (while (and (not (zerop best-time-diff)) (< idx max-idx))
+            (let ((time-diff (vlf-tune-score types idx t
+                                             (+ vlf-tune-load-time
+                                                best-time-diff))))
+              (if time-diff
+                  (progn
+                    (setq time-diff (if (< vlf-tune-load-time time-diff)
+                                        (progn (setq all-less nil)
+                                               (- time-diff
+                                                  vlf-tune-load-time))
+                                      (setq all-more nil)
+                                      (- vlf-tune-load-time time-diff)))
+                    (if (< time-diff best-time-diff)
+                        (setq best-idx idx
+                              best-time-diff time-diff)))
+                (setq all-less nil)))
+            (setq idx (1+ idx)))
+          (* vlf-tune-step (1+ (cond ((or (zerop best-time-diff)
+                                          (eq all-less all-more))
+                                      best-idx)
+                                     (all-less max-idx)
+                                     (t min-idx))))))
+    vlf-batch-size))
+
+(defun vlf-tune-load (types &optional region)
+  "Adjust `vlf-batch-size' slightly to better load time.
+Optimize on TYPES on the nearby REGION.  Use 2 if REGION is nil."
+  (when (eq vlf-tune-enabled t)
+    (or region (setq region 2))
+    (let ((idx (vlf-tune-closest-index vlf-batch-size)))
+      (setq vlf-batch-size (vlf-tune-optimal-load types (- idx region)
+                                                  (+ idx 1 region))))))
+
+(provide 'vlf-tune)
+
+;;; vlf-tune.el ends here

vlf-write.el

@@ -29,49 +29,85 @@
 
 (require 'vlf-base)
 
+(defcustom vlf-save-in-place 'ask
+  "Should VLF save in place when additional adjustment of file content\
+is needed."
+  :group 'vlf :type '(choice (const :tag "Always when applicable" t)
+                             (const :tag "Ask when applicable" 'ask)
+                             (const :tag "Never" nil)))
+
 (defun vlf-write ()
   "Write current chunk to file.  Always return true to disable save.
 If changing size of chunk, shift remaining file content."
   (interactive)
-  (and (buffer-modified-p)
+  (when (and (buffer-modified-p)
-       (or (verify-visited-file-modtime (current-buffer))
+             (or (verify-visited-file-modtime (current-buffer))
-           (y-or-n-p "File has changed since visited or saved.  \
+                 (y-or-n-p "File has changed since visited or saved.\
-Save anyway? "))
+  Save anyway? ")))
-       (if (zerop vlf-file-size)           ;new file
+    (widen)
-           (progn
+    (run-hook-with-args 'vlf-before-batch-functions 'write)
-             (write-region nil nil buffer-file-name vlf-start-pos t)
+    (let ((hexl (derived-mode-p 'hexl-mode)))
-             (setq vlf-file-size (vlf-get-file-size
+      (when hexl
-                                  buffer-file-truename)
+        (if (consp buffer-undo-list)
-                   vlf-end-pos vlf-file-size)
+            (setq buffer-undo-list nil))
-             (vlf-update-buffer-name))
+        (vlf-tune-dehexlify))
-         (widen)
+      (if (zerop vlf-file-size)           ;new file
-         (let* ((region-length (length (encode-coding-region
+          (progn (vlf-tune-write nil nil vlf-start-pos t
-                                        (point-min) (point-max)
+                                 (vlf-tune-encode-length (point-min)
-                                        buffer-file-coding-system t)))
+                                                         (point-max)))
-                (size-change (- vlf-end-pos vlf-start-pos
+                 (if hexl (vlf-tune-hexlify))
-                                region-length)))
+                 (setq vlf-file-size (vlf-get-file-size
-           (if (zerop size-change)
+                                      buffer-file-truename)
-               (write-region nil nil buffer-file-name vlf-start-pos t)
+                       vlf-end-pos vlf-file-size))
-             (let ((tramp-verbose (min 2 tramp-verbose))
+        (let* ((region-length (vlf-tune-encode-length (point-min)
-                   (pos (point))
+                                                      (point-max)))
-                   (font-lock font-lock-mode))
+               (size-change (- vlf-end-pos vlf-start-pos
-               (font-lock-mode 0)
+                               region-length)))
-               (if (< 0 size-change)
+          (if (zerop size-change)
-                   (vlf-file-shift-back size-change)
+              (progn (vlf-tune-write nil nil vlf-start-pos t
-                 (vlf-file-shift-forward (- size-change)))
+                                     (- vlf-end-pos vlf-start-pos))
-               (if font-lock (font-lock-mode 1))
+                     (if hexl (vlf-tune-hexlify)))
-               (vlf-move-to-chunk-2 vlf-start-pos
+            (let ((pos (point))
-                                    (if (< (- vlf-end-pos vlf-start-pos)
+                  (font-lock font-lock-mode)
-                                           vlf-batch-size)
+                  (batch-size vlf-batch-size)
-                                        (+ vlf-start-pos vlf-batch-size)
+                  time)
-                                      vlf-end-pos))
+              (font-lock-mode 0)
-               (vlf-update-buffer-name)
+              (if (or (file-remote-p buffer-file-name)
-               (goto-char pos))))))
+                      (if (eq vlf-save-in-place 'ask)
+                          (y-or-n-p "File content needs be adjusted\
+ till end.  Use temporary copy of the whole file (slower but safer)? ")
+                        (not vlf-save-in-place)))
+                  (let ((file-tmp (make-temp-file "vlf")))
+                    (setq time (float-time))
+                    (copy-file buffer-file-name file-tmp t t t t)
+                    (if (< 0 size-change)
+                        (vlf-file-shift-back size-change region-length
+                                             file-tmp)
+                      (vlf-file-shift-forward (- size-change)
+                                              region-length file-tmp))
+                    (rename-file file-tmp buffer-file-name t))
+                (setq time (float-time))
+                (if (< 0 size-change)
+                    (vlf-file-shift-back size-change region-length)
+                  (vlf-file-shift-forward (- size-change)
+                                          region-length)))
+              (if font-lock (font-lock-mode 1))
+              (setq vlf-batch-size batch-size)
+              (vlf-move-to-chunk-2 vlf-start-pos
+                                   (if (< (- vlf-end-pos vlf-start-pos)
+                                          vlf-batch-size)
+                                       (+ vlf-start-pos vlf-batch-size)
+                                     vlf-end-pos))
+              (goto-char pos)
+              (message "Save took %f seconds" (- (float-time) time)))))))
+    (run-hook-with-args 'vlf-after-batch-functions 'write))
   t)
 
-(defun vlf-file-shift-back (size-change)
+(defun vlf-file-shift-back (size-change write-size &optional file)
-  "Shift file contents SIZE-CHANGE bytes back."
+  "Shift file contents SIZE-CHANGE bytes back.
-  (write-region nil nil buffer-file-name vlf-start-pos t)
+WRITE-SIZE is byte length of saved chunk.
+FILE if given is filename to be used, otherwise `buffer-file-name'."
+  (vlf-tune-write nil nil vlf-start-pos (if file nil t) write-size file)
   (let ((read-start-pos vlf-end-pos)
         (coding-system-for-write 'no-conversion)
         (reporter (make-progress-reporter "Adjusting file content..."
@@ -79,69 +115,84 @@ Save anyway? "))
                                           vlf-file-size)))
     (vlf-with-undo-disabled
      (while (vlf-shift-batch read-start-pos (- read-start-pos
-                                               size-change))
+                                               size-change)
+                             file)
        (setq read-start-pos (+ read-start-pos vlf-batch-size))
        (progress-reporter-update reporter read-start-pos))
      ;; pad end with space
      (erase-buffer)
-     (vlf-verify-size t)
+     (vlf-verify-size t file)
      (insert-char 32 size-change))
-    (write-region nil nil buffer-file-name (- vlf-file-size
+    (vlf-tune-write nil nil (- vlf-file-size size-change)
-                                              size-change) t)
+                    (if file nil t) size-change file)
     (progress-reporter-done reporter)))
 
-(defun vlf-shift-batch (read-pos write-pos)
+(defun vlf-shift-batch (read-pos write-pos file)
   "Read `vlf-batch-size' bytes from READ-POS and write them \
-back at WRITE-POS.  Return nil if EOF is reached, t otherwise."
+back at WRITE-POS using FILE.
+Return nil if EOF is reached, t otherwise."
   (erase-buffer)
-  (vlf-verify-size t)
+  (vlf-verify-size t file)
-  (let ((read-end (+ read-pos vlf-batch-size)))
+  (vlf-tune-batch '(:raw :write) nil file) ;insert speed over temp write file may defer wildly
-    (insert-file-contents-literally buffer-file-name nil
+  (let ((read-end (min (+ read-pos vlf-batch-size) vlf-file-size))) ;compared to the original file
-                                    read-pos
+    (vlf-tune-insert-file-contents-literally read-pos read-end file)
-                                    (min vlf-file-size read-end))
+    (vlf-tune-write nil nil write-pos 0 (- read-end read-pos) file)
-    (write-region nil nil buffer-file-name write-pos 0)
     (< read-end vlf-file-size)))
 
-(defun vlf-file-shift-forward (size-change)
+(defun vlf-file-shift-forward (size-change write-size &optional file)
   "Shift file contents SIZE-CHANGE bytes forward.
+WRITE-SIZE is byte length of saved chunk.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
 Done by saving content up front and then writing previous batch."
-  (let ((read-size (max (/ vlf-batch-size 2) size-change))
+  (vlf-tune-batch '(:raw :write) nil file)
+  (let ((read-size (max vlf-batch-size size-change))
         (read-pos vlf-end-pos)
         (write-pos vlf-start-pos)
         (reporter (make-progress-reporter "Adjusting file content..."
                                           vlf-start-pos
                                           vlf-file-size)))
     (vlf-with-undo-disabled
-     (when (vlf-shift-batches read-size read-pos write-pos t)
+     (when (vlf-shift-batches read-size read-pos write-pos
+                              write-size t file)
+       (vlf-tune-batch '(:raw :write) nil file)
        (setq write-pos (+ read-pos size-change)
-             read-pos (+ read-pos read-size))
+             read-pos (+ read-pos read-size)
+             write-size read-size
+             read-size (max vlf-batch-size size-change))
        (progress-reporter-update reporter write-pos)
        (let ((coding-system-for-write 'no-conversion))
-         (while (vlf-shift-batches read-size read-pos write-pos nil)
+         (while (vlf-shift-batches read-size read-pos write-pos
+                                   write-size nil file)
+           (vlf-tune-batch '(:raw :write) nil file)
            (setq write-pos (+ read-pos size-change)
-                 read-pos (+ read-pos read-size))
+                 read-pos (+ read-pos read-size)
+                 write-size read-size
+                 read-size (max vlf-batch-size size-change))
            (progress-reporter-update reporter write-pos)))))
     (progress-reporter-done reporter)))
 
-(defun vlf-shift-batches (read-size read-pos write-pos hide-read)
+(defun vlf-shift-batches (read-size read-pos write-pos write-size
+                                    hide-read file)
   "Append READ-SIZE bytes of file starting at READ-POS.
 Then write initial buffer content to file at WRITE-POS.
+WRITE-SIZE is byte length of saved chunk.
 If HIDE-READ is non nil, temporarily hide literal read content.
+FILE if given is filename to be used, otherwise `buffer-file-name'.
 Return nil if EOF is reached, t otherwise."
-  (vlf-verify-size t)
+  (vlf-verify-size t file)
   (let ((read-more (< read-pos vlf-file-size))
         (start-write-pos (point-min))
         (end-write-pos (point-max)))
     (when read-more
       (goto-char end-write-pos)
-      (insert-file-contents-literally buffer-file-name nil read-pos
+      (vlf-tune-insert-file-contents-literally
-                                      (min vlf-file-size
+       read-pos (min vlf-file-size (+ read-pos read-size)) file))
-                                           (+ read-pos read-size))))
     ;; write
     (if hide-read ; hide literal region if user has to choose encoding
         (narrow-to-region start-write-pos end-write-pos))
-    (write-region start-write-pos end-write-pos
+    (vlf-tune-write start-write-pos end-write-pos write-pos
-                  buffer-file-name write-pos 0)
+                    (or (and (not read-more) (not file)) 0)
+                    write-size file)
     (delete-region start-write-pos end-write-pos)
     (if hide-read (widen))
     read-more))

vlf.el

@@ -1,13 +1,13 @@
 ;;; vlf.el --- View Large Files  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2006, 2012-2014 Free Software Foundation, Inc.
+;; Copyright (C) 2006, 2012-2015 Free Software Foundation, Inc.
 
-;; Version: 1.4
+;; Version: 1.7
 ;; Keywords: large files, utilities
 ;; Maintainer: Andrey Kotlarski <m00naticus@gmail.com>
 ;; Authors: 2006 Mathias Dahl <mathias.dahl@gmail.com>
 ;;          2012 Sam Steingold <sds@gnu.org>
-;;          2013-2014 Andrey Kotlarski <m00naticus@gmail.com>
+;;          2013-2015 Andrey Kotlarski <m00naticus@gmail.com>
 ;; URL: https://github.com/m00natic/vlfi
 
 ;; This file is free software; you can redistribute it and/or modify
@@ -31,7 +31,7 @@
 ;; which provides several commands for moving around, searching,
 ;; comparing and editing selected part of file.
 ;; To have it offered when opening large files:
-;; (require 'vlf-integrate)
+;; (require 'vlf-setup)
 
 ;; This package was inspired by a snippet posted by Kevin Rodgers,
 ;; showing how to use `insert-file-contents' to extract part of a
@@ -41,12 +41,28 @@
 
 (require 'vlf-base)
 
+(defcustom vlf-before-batch-functions nil
+  "Hook that runs before multiple batch operations.
+One argument is supplied that specifies current action.  Possible
+values are: `write', `ediff', `occur', `search', `goto-line'."
+  :group 'vlf :type 'hook)
+
+(defcustom vlf-after-batch-functions nil
+  "Hook that runs after multiple batch operations.
+One argument is supplied that specifies current action.  Possible
+values are: `write', `ediff', `occur', `search', `goto-line'."
+  :group 'vlf :type 'hook)
+
+(defvar hexl-bits)
+
 (autoload 'vlf-write "vlf-write" "Write current chunk to file." t)
 (autoload 'vlf-re-search-forward "vlf-search"
   "Search forward for REGEXP prefix COUNT number of times." t)
 (autoload 'vlf-re-search-backward "vlf-search"
   "Search backward for REGEXP prefix COUNT number of times." t)
 (autoload 'vlf-goto-line "vlf-search" "Go to line." t)
+(autoload 'vlf-query-replace "vlf-search"
+  "Query replace regexp over whole file." t)
 (autoload 'vlf-occur "vlf-occur"
   "Make whole file occur style index for REGEXP." t)
 (autoload 'vlf-toggle-follow "vlf-follow"
@@ -67,6 +83,7 @@
         (vlf-change-batch-size t)))
     (define-key map "s" 'vlf-re-search-forward)
     (define-key map "r" 'vlf-re-search-backward)
+    (define-key map "%" 'vlf-query-replace)
     (define-key map "o" 'vlf-occur)
     (define-key map "[" 'vlf-beginning-of-file)
     (define-key map "]" 'vlf-end-of-file)
@@ -86,50 +103,81 @@
 
 (define-minor-mode vlf-mode
   "Mode to browse large files in."
-  :lighter " VLF"
+  :group 'vlf :keymap vlf-prefix-map
-  :group 'vlf
+  :lighter (:eval (format " VLF[%d/%d](%s)"
-  :keymap vlf-prefix-map
+                          (/ vlf-end-pos vlf-batch-size)
-  (if vlf-mode
+                          (/ vlf-file-size vlf-batch-size)
-      (progn
+                          (file-size-human-readable vlf-file-size)))
-        (set (make-local-variable 'require-final-newline) nil)
+  (cond (vlf-mode
-        (add-hook 'write-file-functions 'vlf-write nil t)
+         (set (make-local-variable 'require-final-newline) nil)
-        (set (make-local-variable 'revert-buffer-function)
+         (add-hook 'write-file-functions 'vlf-write nil t)
-             'vlf-revert)
+         (set (make-local-variable 'revert-buffer-function)
-        (make-local-variable 'vlf-batch-size)
+              'vlf-revert)
-        (setq vlf-file-size (vlf-get-file-size buffer-file-truename)
+         (make-local-variable 'vlf-batch-size)
-              vlf-start-pos 0
+         (setq vlf-file-size (vlf-get-file-size buffer-file-truename)
-              vlf-end-pos 0)
+               vlf-start-pos 0
-        (let* ((pos (position-bytes (point)))
+               vlf-end-pos 0)
-               (start (* (/ pos vlf-batch-size) vlf-batch-size)))
+         (let* ((pos (position-bytes (point)))
-          (goto-char (byte-to-position (- pos start)))
+                (start (* (/ pos vlf-batch-size) vlf-batch-size)))
-          (vlf-move-to-batch start)))
+           (goto-char (byte-to-position (- pos start)))
-    (kill-local-variable 'revert-buffer-function)
+           (vlf-move-to-batch start))
-    (vlf-stop-follow)
+         (add-hook 'after-change-major-mode-hook 'vlf-keep-alive t t)
-    (when (or (not large-file-warning-threshold)
+         (vlf-keep-alive))
-              (< vlf-file-size large-file-warning-threshold)
+        ((or (not large-file-warning-threshold)
-              (y-or-n-p (format "Load whole file (%s)? "
+             (< vlf-file-size large-file-warning-threshold)
-                                (file-size-human-readable
+             (y-or-n-p (format "Load whole file (%s)? "
-                                 vlf-file-size))))
+                               (file-size-human-readable
-      (kill-local-variable 'require-final-newline)
+                                vlf-file-size))))
-      (remove-hook 'write-file-functions 'vlf-write t)
+         (kill-local-variable 'revert-buffer-function)
-      (let ((pos (+ vlf-start-pos (position-bytes (point)))))
+         (vlf-stop-follow)
-        (vlf-with-undo-disabled
+         (kill-local-variable 'require-final-newline)
-         (insert-file-contents buffer-file-name t nil nil t))
+         (remove-hook 'write-file-functions 'vlf-write t)
-        (goto-char (byte-to-position pos)))
+         (remove-hook 'after-change-major-mode-hook
-      (rename-buffer (file-name-nondirectory buffer-file-name) t))))
+                      'vlf-keep-alive t)
+         (if (derived-mode-p 'hexl-mode)
+             (let ((line (/ (1+ vlf-start-pos) hexl-bits))
+                   (pos (point)))
+               (if (consp buffer-undo-list)
+                   (setq buffer-undo-list nil))
+               (vlf-with-undo-disabled
+                (let ((inhibit-read-only t))
+                  (insert-file-contents-literally buffer-file-name
+                                                  t nil nil t)
+                  (hexlify-buffer)))
+               (set-buffer-modified-p nil)
+               (goto-char (point-min))
+               (forward-line line)
+               (forward-char pos))
+           (let ((pos (+ vlf-start-pos (position-bytes (point))))
+                 (inhibit-read-only t))
+             (vlf-with-undo-disabled
+              (insert-file-contents buffer-file-name t nil nil t))
+             (goto-char (byte-to-position pos)))))
+        (t (setq vlf-mode t))))
+
+(defun vlf-keep-alive ()
+  "Keep `vlf-mode' on major mode change."
+  (if (derived-mode-p 'hexl-mode)
+      (set (make-local-variable 'revert-buffer-function) 'vlf-revert))
+  (setq vlf-mode t))
 
 ;;;###autoload
-(defun vlf (file)
+(defun vlf (file &optional minimal)
-  "View Large FILE in batches.
+  "View Large FILE in batches.  When MINIMAL load just a few bytes.
 You can customize number of bytes displayed by customizing
 `vlf-batch-size'.
 Return newly created buffer."
-  (interactive "fFile to open: ")
+  (interactive (list (read-file-name "File to open: ") nil))
   (let ((vlf-buffer (generate-new-buffer "*vlf*")))
     (set-buffer vlf-buffer)
     (set-visited-file-name file)
     (set-buffer-modified-p nil)
+    (if (or minimal (file-remote-p file))
+        (set (make-local-variable 'vlf-batch-size) 1024))
     (vlf-mode 1)
+    (when minimal                 ;restore batch size to default value
+      (kill-local-variable 'vlf-batch-size)
+      (make-local-variable 'vlf-batch-size))
     (switch-to-buffer vlf-buffer)
     vlf-buffer))
 
@@ -141,6 +189,9 @@ When prefix argument is negative
  append next APPEND number of batches to the existing buffer."
   (interactive "p")
   (vlf-verify-size)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :raw)
+                   '(:insert :encode)))
   (let* ((end (min (+ vlf-end-pos (* vlf-batch-size (abs append)))
                    vlf-file-size))
          (start (if (< append 0)
@@ -157,6 +208,9 @@ When prefix argument is negative
   (interactive "p")
   (if (zerop vlf-start-pos)
       (error "Already at BOF"))
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :raw)
+                   '(:insert :encode)))
   (let* ((start (max 0 (- vlf-start-pos (* vlf-batch-size (abs prepend)))))
          (end (if (< prepend 0)
                   vlf-end-pos
@@ -180,6 +234,45 @@ When prefix argument is negative
              (goto-char (point-max)))
     ad-do-it))
 
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; hexl mode integration
+
+(eval-after-load "hexl"
+  '(progn
+     (defadvice hexl-save-buffer (around vlf-hexl-save
+                                         activate compile)
+       "Prevent hexl save if `vlf-mode' is active."
+       (if vlf-mode
+           (vlf-write)
+         ad-do-it))
+
+     (defadvice hexl-scroll-up (around vlf-hexl-scroll-up
+                                       activate compile)
+       "Slide to next batch if at end of buffer in `vlf-mode'."
+       (if (and vlf-mode (pos-visible-in-window-p (point-max))
+                (or (not (numberp arg)) (< 0 arg)))
+           (progn (vlf-next-batch 1)
+                  (goto-char (point-min)))
+         ad-do-it))
+
+     (defadvice hexl-scroll-down (around vlf-hexl-scroll-down
+                                         activate compile)
+       "Slide to previous batch if at beginning of buffer in `vlf-mode'."
+       (if (and vlf-mode (pos-visible-in-window-p (point-min)))
+           (progn (vlf-prev-batch 1)
+                  (goto-char (point-max)))
+         ad-do-it))
+
+     (defadvice hexl-mode-exit (around vlf-hexl-mode-exit
+                                       activate compile)
+       "Exit `hexl-mode' gracefully in case `vlf-mode' is active."
+       (if (and vlf-mode (not (buffer-modified-p)))
+           (vlf-with-undo-disabled
+            (erase-buffer)
+            ad-do-it
+            (vlf-move-to-chunk-2 vlf-start-pos vlf-end-pos))
+         ad-do-it))))
+
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;; utilities
 
@@ -193,23 +286,34 @@ with the prefix argument DECREASE it is halved."
 
 (defun vlf-set-batch-size (size)
   "Set batch to SIZE bytes and update chunk."
-  (interactive (list (read-number "Size in bytes: " vlf-batch-size)))
+  (interactive
+   (list (read-number "Size in bytes: "
+                      (vlf-tune-optimal-load
+                       (if (derived-mode-p 'hexl-mode)
+                           '(:hexl :raw)
+                         '(:insert :encode))))))
   (setq vlf-batch-size size)
   (vlf-move-to-batch vlf-start-pos))
 
 (defun vlf-beginning-of-file ()
   "Jump to beginning of file content."
   (interactive)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :raw)
+                   '(:insert :encode)))
   (vlf-move-to-batch 0))
 
 (defun vlf-end-of-file ()
   "Jump to end of file content."
   (interactive)
   (vlf-verify-size)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :raw)
+                   '(:insert :encode)))
   (vlf-move-to-batch vlf-file-size))
 
-(defun vlf-revert (&optional _ignore-auto noconfirm)
+(defun vlf-revert (&optional _auto noconfirm)
-  "Revert current chunk.  Ignore _IGNORE-AUTO.
+  "Revert current chunk.  Ignore _AUTO.
 Ask for confirmation if NOCONFIRM is nil."
   (interactive)
   (when (or noconfirm
@@ -221,6 +325,9 @@ Ask for confirmation if NOCONFIRM is nil."
 (defun vlf-jump-to-chunk (n)
   "Go to to chunk N."
   (interactive "nGoto to chunk: ")
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :raw)
+                   '(:insert :encode)))
   (vlf-move-to-batch (* (1- n) vlf-batch-size)))
 
 (defun vlf-no-modifications ()
@@ -229,16 +336,15 @@ Ask for confirmation if NOCONFIRM is nil."
       (error "Save or discard your changes first")
     t))
 
-(defun vlf-move-to-batch (start &optional minimal)
+(defun vlf-move-to-batch (start)
   "Move to batch determined by START.
-Adjust according to file start/end and show `vlf-batch-size' bytes.
+Adjust according to file start/end and show `vlf-batch-size' bytes."
-When given MINIMAL flag, skip non important operations."
   (vlf-verify-size)
   (let* ((start (max 0 start))
          (end (min (+ start vlf-batch-size) vlf-file-size)))
     (if (= vlf-file-size end)          ; re-adjust start
         (setq start (max 0 (- end vlf-batch-size))))
-    (vlf-move-to-chunk start end minimal)))
+    (vlf-move-to-chunk start end)))
 
 (defun vlf-next-batch-from-point ()
   "Display batch of file data starting from current point."