Fix vlf-tune-optimal-load in case best value prunes slower times leaving

all-less flag on.

README.org

@@ -1,27 +1,27 @@
 * View Large Files
 
 Emacs minor mode that allows viewing, editing, searching and comparing
-large files in batches.  Batch size can be adjusted on the fly and
-bounds the memory that is to be used for operations on the file.  This
-way multiple large files (like terabytes or whatever) can be instantly
-and simultaneously accessed without swapping and degraded
-performance.
+large files in batches, trading memory for processor time.  Batch size
+can be adjusted on the fly and bounds the memory that is to be used
+for operations on the file.  This way multiple large files (like
+terabytes or whatever) can be instantly and simultaneously accessed
+without swapping and degraded performance.
 
 This is development version of the GNU ELPA [[http://elpa.gnu.org/packages/vlf][VLF]] package.  Here's what
 it offers in a nutshell:
 
 - regular expression search on whole file (in constant memory
   determined by current batch size)
-- chunk editing (if size has changed, saving is done in constant
-  memory determined by current batch size)
+- chunk editing (save is immediate if size hasn't changed, done in
+  constant memory determined by current batch size otherwise)
 - [[http://www.emacswiki.org/emacs/OccurMode][Occur]] like indexing
 - options to jump to beginning, end or arbitrary file chunk
-- ability to jump/insert given number of batches at once
 - newly added content is acknowledged if file has changed size
   meanwhile
 - automatic scrolling of batches
-- as a minor mode, font locking and functionality of the respective
-  major mode is also present
+- automatic adjustment of batch size for optimal performance
+- as it's a minor mode, font locking and functionality of the
+  respective major mode is also present
 - by batch [[http://www.emacswiki.org/emacs/EdiffMode][Ediff]] comparison
 - can be added as option to automatically open large files
 - smooth integration with [[http://www.emacswiki.org/emacs/HexlMode][hexl-mode]]
@@ -45,19 +45,15 @@ Regular Emacs integers are used, so if you use 32-bit Emacs without
 bignum support, *VLF* will not work with files over 512 MB (maximum
 integer value).
 
-** Memory control
-
-*vlf-batch-size* bounds the memory used for all operations.
-
 * Detail usage
 
 ** Applicability
 
 To have *vlf* offered as choice when opening large files:
 
-#+BEGIN_EXAMPLE
-(require 'vlf-integrate)
-#+END_EXAMPLE
+#+BEGIN_SRC emacs-lisp
+  (require 'vlf-integrate)
+#+END_SRC
 
 You can control when *vlf-mode* is invoked or offered with the
 *vlf-application* customization option.  By default it will offer
@@ -66,10 +62,10 @@ it (you can still call *vlf* command explicitly); to use it without
 asking for large files or to invoke it on all files.  Here's example
 setup such that *vlf-mode* automatically launches for large files:
 
-#+BEGIN_EXAMPLE
-(custom-set-variables
- '(vlf-application 'dont-ask))
-#+END_EXAMPLE
+#+BEGIN_SRC emacs-lisp
+  (custom-set-variables
+   '(vlf-application 'dont-ask))
+#+END_SRC
 
 *** Disable for specific mode
 
@@ -81,32 +77,52 @@ To disable automatic usage of *VLF* for a major mode, add it to
 To disable automatic usage of *VLF* for a function, for example named
 *func* defined in file *file.el*:
 
-#+BEGIN_EXAMPLE
-(vlf-disable-for-function func "file")
-#+END_EXAMPLE
+#+BEGIN_SRC emacs-lisp
+  (vlf-disable-for-function func "file")
+#+END_SRC
 
 ** Keymap
 
 All *VLF* operations are grouped under the *C-c C-v* prefix by
 default.  Here's example how to add another prefix (*C-x v*):
 
-#+BEGIN_EXAMPLE
-(eval-after-load "vlf"
-  '(define-key vlf-prefix-map "\C-xv" vlf-mode-map))
-#+END_EXAMPLE
+#+BEGIN_SRC emacs-lisp
+  (eval-after-load "vlf"
+    '(define-key vlf-prefix-map "\C-xv" vlf-mode-map))
+#+END_SRC
 
-** Control batch size
+** Batch size control
+
+By default *VLF* gathers statistics over how primitive operations
+perform over file and gradually adjusts batch size for better user
+experience.  Operations involving multiple batches are tuned more
+adventurously.  Overall the more jumping around, searching, indexing,
+the better performance should get.
+
+The *vlf-tune-max* option specifies maximum size in bytes a batch
+could eventually get while tuning.
+
+Profiling and tuning can be disabled by:
+
+#+BEGIN_SRC emacs-lisp
+  (custom-set-variables
+     '(vlf-tune-enabled nil))
+#+END_SRC
+
+Or set *vlf-tune-enabled* to '*stats* to profile but not change batch
+size.
 
 Use *M-x vlf-set-batch-size* to change batch size and update chunk
-immediately.
+immediately.  Default size offered is the best according to tune
+statistics so far.
 
 *C-c C-v +* and *C-c C-v -* control current batch size by factors
 of 2.
 
 ** Move around
 
-Scrolling automatically triggers moving to previous or next chunk at
-the beginning or end respectively of the current one.
+Scrolling automatically triggers move to previous or next chunk at the
+beginning or end respectively of the current one.
 
 *C-c C-v n* and *C-c C-v p* move batch by batch.  With positive
 prefix argument they move prefix number of batches.  With negative -
@@ -130,29 +146,28 @@ toggled with *C-c C-v f*.
 ** Search whole file
 
 *C-c C-v s* and *C-c C-v r* search forward and backward respectively
-over the whole file.  This is done batch by batch so if you have
-really huge file - you'd better set somewhat bigger batch size
-beforehand.
+over the whole file, batch by batch.
 
 ** Occur over whole file
 
-*C-c C-v o* builds index for given regular expression just like M-x
-occur*.  It does so batch by batch over the whole file.  Note that
-even if you prematurely stop it with *C-g*, it will still show index
-of what's found so far.
+*C-c C-v o* builds index over whole file for given regular expression
+just like *M-x occur*.  Note that even if you prematurely stop it with
+*C-g*, it will still show what's found so far.
+
+Result buffer uses *vlf-occur-mode* which allows to optionally open
+new *VLF* buffer on jump to match (using *C-u* before hitting RET or
+*o*), thus having multiple simultaneous views of the same file.  Also
+results can be serialized to file for later reuse.
 
 ** Jump to line
 
-*C-c C-v l* jumps to given line in file.  This is done by searching
-from the beginning, so again the bigger current batch size, the
-quicker.  With negative argument, lines are counted from the end of
-file.
+*C-c C-v l* jumps to given line in file.  With negative argument,
+lines are counted from the end of file.
 
 ** Edit and save
 
 If editing doesn't change size of the chunk, only this chunk is saved.
-Otherwise the remaining part of the file is adjusted batch by batch,
-so again you'd better have bigger current batch size.
+Otherwise the remaining part of the file is adjusted batch by batch.
 
 ** By batch Ediff
 
@@ -174,6 +189,6 @@ A couple of hooks are run whenever updating chunk:
 
 Some operations may trigger multiple chunk moves.  There are a couple
 of hooks that run in such cases: *vlf-before-batch-functions* and
-*vlf-after-batch-functions*.  They are passed one argument that
+*vlf-after-batch-functions*.  They are passed one argument which
 specifies type of operation that runs.  Possible values are the
 symbols: *write*, *ediff*, *occur*, *search* and *goto-line*.

vlf-base.el

@@ -27,10 +27,7 @@
 
 ;;; Code:
 
-(defcustom vlf-batch-size 1024
-  "Defines how large each batch of file data is (in bytes)."
-  :group 'vlf :type 'integer)
-(put 'vlf-batch-size 'permanent-local t)
+(require 'vlf-tune)
 
 (defcustom vlf-before-chunk-update nil
   "Hook that runs before chunk update."
@@ -50,10 +47,6 @@
 (make-variable-buffer-local 'vlf-end-pos)
 (put 'vlf-end-pos 'permanent-local t)
 
-(defvar vlf-file-size 0 "Total size of presented file.")
-(make-variable-buffer-local 'vlf-file-size)
-(put 'vlf-file-size 'permanent-local t)
-
 (defconst vlf-sample-size 24
   "Minimal number of bytes that can be properly decoded.")
 
@@ -108,8 +101,7 @@ bytes added to the end."
                           0)))
              (setq vlf-start-pos place
                    vlf-end-pos place)
-             (if (not minimal)
-                 (vlf-update-buffer-name))
+             (or minimal (vlf-update-buffer-name))
              (cons (- start place) (- place end)))))
         ((or (/= start vlf-start-pos)
              (/= end vlf-end-pos))
@@ -126,85 +118,112 @@ bytes added to the end."
   (let* ((modified (buffer-modified-p))
          (start (max 0 start))
          (end (min end vlf-file-size))
+         (hexl (derived-mode-p 'hexl-mode))
+         restore-hexl hexl-undo-list
          (edit-end (if modified
-                       (+ vlf-start-pos
-                          (length (encode-coding-region
-                                   (point-min) (point-max)
-                                   buffer-file-coding-system t)))
-                     vlf-end-pos)))
-    (cond
-     ((or (< edit-end start) (< end vlf-start-pos)
-          (not (verify-visited-file-modtime (current-buffer))))
-      (when (or (not modified)
-                (y-or-n-p "Chunk modified, are you sure? ")) ;full chunk renewal
-        (set-buffer-modified-p nil)
-        (vlf-move-to-chunk-2 start end)))
-     ((and (= start vlf-start-pos) (= end edit-end))
-      (or modified (vlf-move-to-chunk-2 start end)))
-     ((or (and (<= start vlf-start-pos) (<= edit-end end))
-          (not modified)
-          (y-or-n-p "Chunk modified, are you sure? "))
-      (run-hooks 'vlf-before-chunk-update)
-      (let ((shift-start 0)
-            (shift-end 0))
-        (let ((pos (+ (position-bytes (point)) vlf-start-pos))
-              (inhibit-read-only t))
-          (cond ((= end vlf-start-pos)
-                 (or (eq buffer-undo-list t)
-                     (setq buffer-undo-list nil))
-                 (vlf-with-undo-disabled (erase-buffer))
-                 (setq modified nil))
-                ((< end edit-end)
-                 (setq end (car (vlf-delete-region
-                                 (point-min) vlf-start-pos edit-end
-                                 end (min (or (byte-to-position
-                                               (- end vlf-start-pos))
-                                              (point-min))
-                                          (point-max))
-                                 nil))))
-                ((< edit-end end)
-                 (vlf-with-undo-disabled
-                  (setq shift-end (cdr (vlf-insert-file-contents
-                                        vlf-end-pos end nil t
-                                        (point-max)))))))
-          (setq vlf-end-pos (+ end shift-end))
-          (cond ((= start edit-end)
-                 (or (eq buffer-undo-list t)
-                     (setq buffer-undo-list nil))
-                 (vlf-with-undo-disabled
-                  (delete-region (point-min) (point)))
-                 (setq modified nil))
-                ((< vlf-start-pos start)
-                 (let ((del-info (vlf-delete-region
-                                  (point-min) vlf-start-pos
-                                  vlf-end-pos start
-                                  (min (or (byte-to-position
-                                            (- start vlf-start-pos))
-                                           (point))
-                                       (point-max)) t)))
-                   (setq start (car del-info))
-                   (vlf-shift-undo-list (- (point-min)
-                                           (cdr del-info)))))
-                ((< start vlf-start-pos)
-                 (let ((edit-end-pos (point-max)))
-                   (vlf-with-undo-disabled
-                    (setq shift-start (car (vlf-insert-file-contents
-                                            start vlf-start-pos t nil
-                                            edit-end-pos)))
-                    (goto-char (point-min))
-                    (insert (delete-and-extract-region
-                             edit-end-pos (point-max))))
-                   (vlf-shift-undo-list (- (point-max)
-                                           edit-end-pos)))))
-          (setq start (- start shift-start))
-          (goto-char (or (byte-to-position (- pos start))
-                         (byte-to-position (- pos vlf-start-pos))
-                         (point-max)))
-          (setq vlf-start-pos start))
-        (set-buffer-modified-p modified)
-        (set-visited-file-modtime)
-        (run-hooks 'vlf-after-chunk-update)
-        (cons shift-start shift-end))))))
+                       (progn
+                         (when hexl
+                           (setq restore-hexl t
+                                 hexl-undo-list buffer-undo-list
+                                 buffer-undo-list t)
+                           (vlf-tune-dehexlify))
+                         (+ vlf-start-pos
+                            (vlf-tune-encode-length (point-min)
+                                                    (point-max))))
+                     vlf-end-pos))
+         (shifts
+          (cond
+           ((or (< edit-end start) (< end vlf-start-pos)
+                (not (verify-visited-file-modtime (current-buffer))))
+            (when (or (not modified)
+                      (y-or-n-p "Chunk modified, are you sure? ")) ;full chunk renewal
+              (set-buffer-modified-p nil)
+              (if (consp hexl-undo-list)
+                  (setq hexl-undo-list nil))
+              (vlf-move-to-chunk-2 start end)))
+           ((and (= start vlf-start-pos) (= end edit-end))
+            (unless modified
+              (if (consp hexl-undo-list)
+                  (setq hexl-undo-list nil))
+              (vlf-move-to-chunk-2 start end)))
+           ((or (and (<= start vlf-start-pos) (<= edit-end end))
+                (not modified)
+                (y-or-n-p "Chunk modified, are you sure? "))
+            (run-hooks 'vlf-before-chunk-update)
+            (when (and hexl (not restore-hexl))
+              (if (consp buffer-undo-list)
+                  (setq buffer-undo-list nil))
+              (vlf-tune-dehexlify))
+            (let ((shift-start 0)
+                  (shift-end 0))
+              (let ((pos (+ (position-bytes (point)) vlf-start-pos))
+                    (inhibit-read-only t))
+                (cond ((= end vlf-start-pos)
+                       (or (eq buffer-undo-list t)
+                           (setq buffer-undo-list nil))
+                       (vlf-with-undo-disabled (erase-buffer))
+                       (setq modified nil))
+                      ((< end edit-end)
+                       (setq end (car (vlf-delete-region
+                                       (point-min) vlf-start-pos
+                                       edit-end end
+                                       (min (or (byte-to-position
+                                                 (- end vlf-start-pos))
+                                                (point-min))
+                                            (point-max))
+                                       nil))))
+                      ((< edit-end end)
+                       (vlf-with-undo-disabled
+                        (setq shift-end (cdr (vlf-insert-file-contents
+                                              vlf-end-pos end nil t
+                                              (point-max)))))))
+                (setq vlf-end-pos (+ end shift-end))
+                (cond ((= start edit-end)
+                       (or (eq buffer-undo-list t)
+                           (setq buffer-undo-list nil))
+                       (vlf-with-undo-disabled
+                        (delete-region (point-min) (point)))
+                       (setq modified nil))
+                      ((< vlf-start-pos start)
+                       (let ((del-info (vlf-delete-region
+                                        (point-min) vlf-start-pos
+                                        vlf-end-pos start
+                                        (min (or
+                                              (byte-to-position
+                                               (- start vlf-start-pos))
+                                              (point))
+                                             (point-max)) t)))
+                         (setq start (car del-info))
+                         (vlf-shift-undo-list (- (point-min)
+                                                 (cdr del-info)))))
+                      ((< start vlf-start-pos)
+                       (let ((edit-end-pos (point-max)))
+                         (vlf-with-undo-disabled
+                          (setq shift-start (car
+                                             (vlf-insert-file-contents
+                                              start vlf-start-pos t nil
+                                              edit-end-pos)))
+                          (goto-char (point-min))
+                          (insert (delete-and-extract-region
+                                   edit-end-pos (point-max))))
+                         (vlf-shift-undo-list (- (point-max)
+                                                 edit-end-pos)))))
+                (setq start (- start shift-start))
+                (goto-char (or (byte-to-position (- pos start))
+                               (byte-to-position (- pos vlf-start-pos))
+                               (point-max)))
+                (setq vlf-start-pos start))
+              (set-buffer-modified-p modified)
+              (set-visited-file-modtime)
+              (when hexl
+                (vlf-tune-hexlify)
+                (setq restore-hexl nil))
+              (run-hooks 'vlf-after-chunk-update)
+              (cons shift-start shift-end))))))
+    (when restore-hexl
+      (vlf-tune-hexlify)
+      (setq buffer-undo-list hexl-undo-list))
+    shifts))
 
 (defun vlf-move-to-chunk-2 (start end)
   "Unconditionally move to chunk enclosed by START END bytes.
@@ -218,11 +237,14 @@ bytes added to the end."
     (let ((inhibit-read-only t)
           (pos (position-bytes (point))))
       (vlf-with-undo-disabled
-       (erase-buffer)
-       (setq shifts (vlf-insert-file-contents vlf-start-pos
-                                              vlf-end-pos t t)
-             vlf-start-pos (- vlf-start-pos (car shifts))
-             vlf-end-pos (+ vlf-end-pos (cdr shifts)))
+       (let ((hexl (derived-mode-p 'hexl-mode)))
+         (if hexl (hexl-mode-exit t))
+         (erase-buffer)
+         (setq shifts (vlf-insert-file-contents vlf-start-pos
+                                                vlf-end-pos t t)
+               vlf-start-pos (- vlf-start-pos (car shifts))
+               vlf-end-pos (+ vlf-end-pos (cdr shifts)))
+         (if hexl (vlf-tune-hexlify)))
        (goto-char (or (byte-to-position (+ pos (car shifts)))
                       (point-max)))))
     (set-buffer-modified-p nil)
@@ -263,7 +285,7 @@ bytes added to the end."
 
 (defun vlf-insert-file-contents-1 (start end)
   "Extract decoded file bytes START to END."
-  (insert-file-contents buffer-file-name nil start end))
+  (vlf-tune-insert-file-contents start end))
 
 (defun vlf-adjust-start (start end position adjust-end)
   "Adjust chunk beginning at absolute START to END till content can\
@@ -315,12 +337,10 @@ which deletion was performed."
                               (eq encode-direction 'end)
                             (< (- end border) (- border start))))
          (dist (if encode-from-end
-                   (- end (length (encode-coding-region
-                                   cut-point (point-max)
-                                   buffer-file-coding-system t)))
-                 (+ start (length (encode-coding-region
-                                   position cut-point
-                                   buffer-file-coding-system t)))))
+                   (- end (vlf-tune-encode-length cut-point
+                                                  (point-max)))
+                 (+ start (vlf-tune-encode-length position
+                                                  cut-point))))
          (len 0))
     (if (< border dist)
         (while (< border dist)
@@ -348,7 +368,7 @@ which deletion was performed."
 
 (defun vlf-shift-undo-list (n)
   "Shift undo list element regions by N."
-  (or (eq buffer-undo-list t)
+  (or (null buffer-undo-list) (eq buffer-undo-list t)
       (setq buffer-undo-list
             (nreverse
              (let ((min (point-min))

vlf-ediff.el

@@ -34,7 +34,6 @@
   "If non nil, specifies that ediff is done over VLF buffers.")
 (make-variable-buffer-local 'vlf-ediff-session)
 
-;;;###autoload
 (defun vlf-ediff-buffers (buffer-A buffer-B)
   "Run batch by batch ediff over VLF buffers BUFFER-A and BUFFER-B.
 Batch size is determined by the size in BUFFER-A.
@@ -93,10 +92,10 @@ respectively of difference list, runs ediff over the adjacent chunks."
                                       dir-B)))
                                    (ediff-get-default-file-name f 1)))
            (read-number "Batch size (in bytes): " vlf-batch-size))))
-  (let ((buffer-A (vlf file-A)))
+  (let ((buffer-A (vlf file-A t)))
     (set-buffer buffer-A)
     (vlf-set-batch-size batch-size)
-    (let ((buffer-B (vlf file-B)))
+    (let ((buffer-B (vlf file-B t)))
       (vlf-ediff-buffers buffer-A buffer-B))))
 
 (defadvice ediff-next-difference (around vlf-ediff-next-difference
@@ -161,12 +160,14 @@ logical chunks in case there is no difference at the current ones."
         (point-max-A (point-max))
         (font-lock-A font-lock-mode)
         (min-file-size vlf-file-size)
-        (forward-p (eq next-func 'vlf-next-chunk)))
+        (forward-p (eq next-func 'vlf-next-chunk))
+        (is-hexl (derived-mode-p 'hexl-mode)))
     (font-lock-mode 0)
     (set-buffer buffer-B)
     (run-hook-with-args 'vlf-before-batch-functions 'ediff)
     (setq buffer-B (current-buffer)
-          min-file-size (min min-file-size vlf-file-size))
+          min-file-size (min min-file-size vlf-file-size)
+          is-hexl (or is-hexl (derived-mode-p 'hexl-mode)))
     (let ((tramp-verbose (if (boundp 'tramp-verbose)
                              (min tramp-verbose 2)))
           (end-B (= vlf-start-pos vlf-end-pos))
@@ -187,7 +188,7 @@ logical chunks in case there is no difference at the current ones."
                                     buffer-B (point-min) (point-max)))
                             (with-current-buffer ediff-buffer
                               (ediff-update-diffs)
-                              (and (not end-A) (not end-B)
+                              (and (not end-A) (not end-B) (not is-hexl)
                                    (vlf-ediff-refine buffer-A
                                                      buffer-B))
                               (zerop ediff-number-of-differences))))
@@ -221,9 +222,10 @@ logical chunks in case there is no difference at the current ones."
                 (vlf-beginning-of-file))
               (set-buffer ediff-buffer)
               (ediff-update-diffs)
-              (if (or (not forward-p)
-                      (and (not end-A) (not end-B)))
-                  (vlf-ediff-refine buffer-A buffer-B)))
+              (or is-hexl
+                  (if (or (not forward-p)
+                          (and (not end-A) (not end-B)))
+                      (vlf-ediff-refine buffer-A buffer-B))))
             (setq done t))
         (unless done
           (set-buffer buffer-A)
@@ -234,7 +236,8 @@ logical chunks in case there is no difference at the current ones."
           (vlf-move-to-chunk (car chunk-B) (cdr chunk-B))
           (set-buffer ediff-buffer)
           (ediff-update-diffs)
-          (vlf-ediff-refine buffer-A buffer-B))
+          (or is-hexl
+              (vlf-ediff-refine buffer-A buffer-B)))
         (set-buffer buffer-A)
         (if font-lock-A (font-lock-mode 1))
         (run-hook-with-args 'vlf-after-batch-functions 'ediff)

vlf-integrate.el

@@ -29,6 +29,10 @@
 (defgroup vlf nil "View Large Files in Emacs."
   :prefix "vlf-" :group 'files)
 
+(defcustom vlf-batch-size 1000000
+  "Defines how large each batch of file data initially is (in bytes)."
+  :group 'vlf :type 'integer)
+
 (defcustom vlf-application 'ask
   "Determines when `vlf' will be offered on opening files.
 Possible values are: nil to never use it;
@@ -98,7 +102,8 @@ OP-TYPE specifies the file operation being performed over FILENAME."
     (vlf filename)
     (error ""))
    ((and large-file-warning-threshold
-         (< large-file-warning-threshold size))
+         (< large-file-warning-threshold size)
+         (< vlf-batch-size size))
     (if (eq vlf-application 'dont-ask)
         (progn (vlf filename)
                (error ""))

vlf-occur.el

@@ -29,6 +29,21 @@
 
 (require 'vlf)
 
+(defvar vlf-occur-vlf-file nil "VLF file that is searched.")
+(make-variable-buffer-local 'vlf-occur-vlf-file)
+
+(defvar vlf-occur-vlf-buffer nil "VLF buffer that is scanned.")
+(make-variable-buffer-local 'vlf-occur-vlf-buffer)
+
+(defvar vlf-occur-regexp)
+(make-variable-buffer-local 'vlf-occur-regexp)
+
+(defvar vlf-occur-hexl nil "Is `hexl-mode' active?")
+(make-variable-buffer-local 'vlf-occur-hexl)
+
+(defvar vlf-occur-lines 0 "Number of lines scanned by `vlf-occur'.")
+(make-variable-buffer-local 'vlf-occur-lines)
+
 (defvar vlf-occur-mode-map
   (let ((map (make-sparse-keymap)))
     (define-key map "n" 'vlf-occur-next-match)
@@ -37,16 +52,18 @@
     (define-key map "\M-\r" 'vlf-occur-visit-new-buffer)
     (define-key map [mouse-1] 'vlf-occur-visit)
     (define-key map "o" 'vlf-occur-show)
+    (define-key map [remap save-buffer] 'vlf-occur-save)
     map)
   "Keymap for command `vlf-occur-mode'.")
 
 (define-derived-mode vlf-occur-mode special-mode "VLF[occur]"
-  "Major mode for showing occur matches of VLF opened files.")
+  "Major mode for showing occur matches of VLF opened files."
+  (add-hook 'write-file-functions 'vlf-occur-save nil t))
 
 (defun vlf-occur-next-match ()
   "Move cursor to next match."
   (interactive)
-  (if (eq (get-char-property (point) 'face) 'match)
+  (if (eq (get-text-property (point) 'face) 'match)
       (goto-char (next-single-property-change (point) 'face)))
   (goto-char (or (text-property-any (point) (point-max) 'face 'match)
                  (text-property-any (point-min) (point)
@@ -55,9 +72,9 @@
 (defun vlf-occur-prev-match ()
   "Move cursor to previous match."
   (interactive)
-  (if (eq (get-char-property (point) 'face) 'match)
+  (if (eq (get-text-property (point) 'face) 'match)
       (goto-char (previous-single-property-change (point) 'face)))
-  (while (not (eq (get-char-property (point) 'face) 'match))
+  (while (not (eq (get-text-property (point) 'face) 'match))
     (goto-char (or (previous-single-property-change (point) 'face)
                    (point-max)))))
 
@@ -91,84 +108,140 @@ EVENT may hold details of the invocation."
     (goto-char (posn-point (event-end event))))
   (let* ((pos (point))
          (pos-relative (- pos (line-beginning-position) 1))
-         (file (get-char-property pos 'file)))
-    (if file
-        (let ((chunk-start (get-char-property pos 'chunk-start))
-              (chunk-end (get-char-property pos 'chunk-end))
-              (vlf-buffer (get-char-property pos 'buffer))
+         (chunk-start (get-text-property pos 'chunk-start)))
+    (if chunk-start
+        (let ((chunk-end (get-text-property pos 'chunk-end))
+              (file (if (file-exists-p vlf-occur-vlf-file)
+                        vlf-occur-vlf-file
+                      (setq vlf-occur-vlf-file
+                            (read-file-name
+                             (concat vlf-occur-vlf-file
+                                     " doesn't exist, locate it: ")))))
+              (vlf-buffer vlf-occur-vlf-buffer)
+              (not-hexl (not vlf-occur-hexl))
               (occur-buffer (current-buffer))
-              (match-pos (+ (get-char-property pos 'line-pos)
+              (match-pos (+ (get-text-property pos 'line-pos)
                             pos-relative)))
           (cond (current-prefix-arg
-                 (setq vlf-buffer (vlf file))
+                 (setq vlf-buffer (vlf file t))
+                 (or not-hexl (vlf-tune-hexlify))
                  (switch-to-buffer occur-buffer))
                 ((not (buffer-live-p vlf-buffer))
-                 (or (catch 'found
-                       (dolist (buf (buffer-list))
-                         (set-buffer buf)
-                         (and vlf-mode (equal file buffer-file-name)
-                              (setq vlf-buffer buf)
-                              (throw 'found t))))
-                     (setq vlf-buffer (vlf file)))
-                 (switch-to-buffer occur-buffer)))
+                 (unless (catch 'found
+                           (dolist (buf (buffer-list))
+                             (set-buffer buf)
+                             (and vlf-mode
+                                  (equal file buffer-file-name)
+                                  (eq (not (derived-mode-p 'hexl-mode))
+                                      not-hexl)
+                                  (setq vlf-buffer buf)
+                                  (throw 'found t))))
+                   (setq vlf-buffer (vlf file t))
+                   (or not-hexl (vlf-tune-hexlify)))
+                 (switch-to-buffer occur-buffer)
+                 (setq vlf-occur-vlf-buffer vlf-buffer)))
           (pop-to-buffer vlf-buffer)
           (vlf-move-to-chunk chunk-start chunk-end)
           (goto-char match-pos)))))
 
+(defun vlf-occur-other-buffer (regexp)
+  "Make whole file occur style index for REGEXP branching to new buffer.
+Prematurely ending indexing will still show what's found so far."
+  (let ((vlf-buffer (current-buffer))
+        (file buffer-file-name)
+        (batch-size vlf-batch-size)
+        (is-hexl (derived-mode-p 'hexl-mode))
+        (insert-bps vlf-tune-insert-bps)
+        (encode-bps vlf-tune-encode-bps)
+        (hexl-bps vlf-tune-hexl-bps)
+        (dehexlify-bps vlf-tune-dehexlify-bps))
+    (with-temp-buffer
+      (setq buffer-file-name file
+            buffer-file-truename file
+            buffer-undo-list t)
+      (set-buffer-modified-p nil)
+      (set (make-local-variable 'vlf-batch-size) batch-size)
+      (when vlf-tune-enabled
+        (setq vlf-tune-insert-bps insert-bps
+              vlf-tune-encode-bps encode-bps)
+        (if is-hexl
+            (progn (setq vlf-tune-hexl-bps hexl-bps
+                         vlf-tune-dehexlify-bps dehexlify-bps)
+                   (vlf-tune-batch '(:hexl :dehexlify :insert :encode)))
+          (vlf-tune-batch '(:insert :encode))))
+      (vlf-mode 1)
+      (if is-hexl (vlf-tune-hexlify))
+      (goto-char (point-min))
+      (vlf-with-undo-disabled
+       (vlf-build-occur regexp vlf-buffer))
+      (when vlf-tune-enabled
+        (setq insert-bps vlf-tune-insert-bps
+              encode-bps vlf-tune-encode-bps)
+        (if is-hexl
+            (setq insert-bps vlf-tune-insert-bps
+                  encode-bps vlf-tune-encode-bps))))
+    (when vlf-tune-enabled              ;merge back tune measurements
+      (setq vlf-tune-insert-bps insert-bps
+            vlf-tune-encode-bps encode-bps)
+      (if is-hexl
+          (setq vlf-tune-insert-bps insert-bps
+                vlf-tune-encode-bps encode-bps)))))
+
 (defun vlf-occur (regexp)
   "Make whole file occur style index for REGEXP.
 Prematurely ending indexing will still show what's found so far."
   (interactive (list (read-regexp "List lines matching regexp"
                                   (if regexp-history
                                       (car regexp-history)))))
-  (if (buffer-modified-p) ;use temporary buffer not to interfere with modifications
-      (let ((vlf-buffer (current-buffer))
-            (file buffer-file-name)
-            (batch-size vlf-batch-size))
-        (with-temp-buffer
-          (setq buffer-file-name file)
-          (set-buffer-modified-p nil)
-          (set (make-local-variable 'vlf-batch-size) batch-size)
-          (vlf-mode 1)
-          (goto-char (point-min))
-          (run-hook-with-args 'vlf-before-batch-functions 'occur)
-          (vlf-with-undo-disabled
-           (vlf-build-occur regexp vlf-buffer))
-          (run-hook-with-args 'vlf-after-batch-functions 'occur)))
-    (run-hook-with-args 'vlf-before-batch-functions 'occur)
+  (run-hook-with-args 'vlf-before-batch-functions 'occur)
+  (if (or (buffer-modified-p)
+          (< vlf-batch-size vlf-start-pos))
+      (vlf-occur-other-buffer regexp)
     (let ((start-pos vlf-start-pos)
           (end-pos vlf-end-pos)
-          (pos (point)))
+          (pos (point))
+          (batch-size vlf-batch-size)
+          (is-hexl (derived-mode-p 'hexl-mode)))
+      (vlf-tune-batch (if (derived-mode-p 'hexl-mode)
+                          '(:hexl :dehexlify :insert :encode)
+                        '(:insert :encode)))
       (vlf-with-undo-disabled
-       (vlf-beginning-of-file)
+       (vlf-move-to-batch 0)
        (goto-char (point-min))
        (unwind-protect (vlf-build-occur regexp (current-buffer))
          (vlf-move-to-chunk start-pos end-pos)
-         (goto-char pos))))
-    (run-hook-with-args 'vlf-after-batch-functions 'occur)))
+         (if is-hexl (vlf-tune-hexlify))
+         (goto-char pos)
+         (setq vlf-batch-size batch-size)))))
+  (run-hook-with-args 'vlf-after-batch-functions 'occur))
 
 (defun vlf-build-occur (regexp vlf-buffer)
   "Build occur style index for REGEXP over VLF-BUFFER."
-  (let ((tramp-verbose (if (boundp 'tramp-verbose)
-                           (min tramp-verbose 2)))
-        (case-fold-search t)
-        (line 1)
-        (last-match-line 0)
-        (last-line-pos (point-min))
-        (file buffer-file-name)
-        (total-matches 0)
-        (match-end-pos (+ vlf-start-pos (position-bytes (point))))
-        (occur-buffer (generate-new-buffer
-                       (concat "*VLF-occur " (file-name-nondirectory
-                                              buffer-file-name)
-                               "*")))
-        (line-regexp (concat "\\(?5:[\n\C-m]\\)\\|\\(?10:"
-                             regexp "\\)"))
-        (batch-step (/ vlf-batch-size 8))
-        (end-of-file nil)
-        (reporter (make-progress-reporter
-                   (concat "Building index for " regexp "...")
-                   vlf-start-pos vlf-file-size)))
+  (let* ((tramp-verbose (if (boundp 'tramp-verbose)
+                            (min tramp-verbose 2)))
+         (case-fold-search t)
+         (line 1)
+         (last-match-line 0)
+         (last-line-pos (point-min))
+         (total-matches 0)
+         (match-end-pos (+ vlf-start-pos (position-bytes (point))))
+         (occur-buffer (generate-new-buffer
+                        (concat "*VLF-occur " (file-name-nondirectory
+                                               buffer-file-name)
+                                "*")))
+         (line-regexp (concat "\\(?5:[\n\C-m]\\)\\|\\(?10:"
+                              regexp "\\)"))
+         (batch-step (min 1024 (/ vlf-batch-size 8)))
+         (is-hexl (derived-mode-p 'hexl-mode))
+         (end-of-file nil)
+         (time (float-time))
+         (tune-types (if is-hexl '(:hexl :dehexlify :insert :encode)
+                       '(:insert :encode)))
+         (reporter (make-progress-reporter
+                    (concat "Building index for " regexp "...")
+                    vlf-start-pos vlf-file-size)))
+    (with-current-buffer occur-buffer
+      (setq buffer-undo-list t))
     (unwind-protect
         (progn
           (while (not end-of-file)
@@ -197,8 +270,6 @@ Prematurely ending indexing will still show what's found so far."
                                           (number-to-string line)
                                           'face 'shadow)))
                           (insert (propertize line-text ; insert line
-                                              'file file
-                                              'buffer vlf-buffer
                                               'chunk-start chunk-start
                                               'chunk-end chunk-end
                                               'mouse-face '(highlight)
@@ -221,35 +292,146 @@ Prematurely ending indexing will still show what's found so far."
                                          total-matches))))))))
               (setq end-of-file (= vlf-end-pos vlf-file-size))
               (unless end-of-file
+                (vlf-tune-batch tune-types)
                 (let ((batch-move (- vlf-end-pos batch-step)))
-                  (vlf-move-to-batch (if (< batch-move match-end-pos)
-                                         match-end-pos
-                                       batch-move) t))
-                (goto-char (if (< vlf-start-pos match-end-pos)
-                               (or (byte-to-position (- match-end-pos
-                                                        vlf-start-pos))
-                                   (point-min))
-                             (point-min)))
+                  (vlf-move-to-batch (if (or is-hexl
+                                             (< match-end-pos
+                                                batch-move))
+                                         batch-move
+                                       match-end-pos) t))
+                (goto-char (if (or is-hexl
+                                   (<= match-end-pos vlf-start-pos))
+                               (point-min)
+                             (or (byte-to-position (- match-end-pos
+                                                      vlf-start-pos))
+                                 (point-min))))
                 (setq last-match-line 0
                       last-line-pos (line-beginning-position))
                 (progress-reporter-update reporter vlf-end-pos))))
           (progress-reporter-done reporter))
       (set-buffer-modified-p nil)
       (if (zerop total-matches)
-          (progn (with-current-buffer occur-buffer
-                   (set-buffer-modified-p nil))
-                 (kill-buffer occur-buffer)
-                 (message "No matches for \"%s\"" regexp))
-        (with-current-buffer occur-buffer
-          (goto-char (point-min))
-          (insert (propertize
-                   (format "%d matches in %d lines for \"%s\" \
+          (progn (kill-buffer occur-buffer)
+                 (message "No matches for \"%s\" (%f secs)"
+                          regexp (- (float-time) time)))
+        (let ((file buffer-file-name)
+              (dir default-directory))
+          (with-current-buffer occur-buffer
+            (goto-char (point-min))
+            (insert (propertize
+                     (format "%d matches from %d lines for \"%s\" \
 in file: %s" total-matches line regexp file)
-                   'face 'underline))
-          (set-buffer-modified-p nil)
-          (forward-char 2)
-          (vlf-occur-mode))
-        (display-buffer occur-buffer)))))
+                     'face 'underline))
+            (set-buffer-modified-p nil)
+            (forward-char 2)
+            (vlf-occur-mode)
+            (setq default-directory dir
+                  vlf-occur-vlf-file file
+                  vlf-occur-vlf-buffer vlf-buffer
+                  vlf-occur-regexp regexp
+                  vlf-occur-hexl is-hexl
+                  vlf-occur-lines line)))
+        (display-buffer occur-buffer)
+        (message "Occur finished for \"%s\" (%f secs)"
+                 regexp (- (float-time) time))))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; save, load vlf-occur data
+
+(defun vlf-occur-save (file)
+  "Serialize `vlf-occur' results to FILE which can later be reloaded."
+  (interactive (list (or buffer-file-name
+                         (read-file-name "Save vlf-occur results in: "
+                                         nil nil nil
+                                         (concat
+                                          (file-name-nondirectory
+                                           vlf-occur-vlf-file)
+                                          ".vlfo")))))
+  (setq buffer-file-name file)
+  (let ((vlf-occur-save-buffer
+         (generate-new-buffer (concat "*VLF-occur-save "
+                                      (file-name-nondirectory file)
+                                      "*"))))
+    (with-current-buffer vlf-occur-save-buffer
+      (setq buffer-file-name file
+            buffer-undo-list t)
+      (insert ";; -*- eval: (vlf-occur-load) -*-\n"))
+    (prin1 (list vlf-occur-vlf-file vlf-occur-regexp vlf-occur-hexl
+                 vlf-occur-lines)
+           vlf-occur-save-buffer)
+    (save-excursion
+      (goto-char (point-min))
+      (while (zerop (forward-line))
+        (let* ((pos (1+ (point)))
+               (line (get-char-property (1- pos) 'before-string)))
+          (if line
+              (prin1 (list (string-to-number line)
+                           (get-text-property pos 'chunk-start)
+                           (get-text-property pos 'chunk-end)
+                           (get-text-property pos 'line-pos)
+                           (buffer-substring-no-properties
+                            pos (line-end-position)))
+                     vlf-occur-save-buffer)))))
+    (with-current-buffer vlf-occur-save-buffer
+      (save-buffer))
+    (kill-buffer vlf-occur-save-buffer))
+  t)
+
+;;;###autoload
+(defun vlf-occur-load ()
+  "Load serialized `vlf-occur' results from current buffer."
+  (interactive)
+  (goto-char (point-min))
+  (let* ((vlf-occur-data-buffer (current-buffer))
+         (header (read vlf-occur-data-buffer))
+         (vlf-file (nth 0 header))
+         (regexp (nth 1 header))
+         (all-lines (nth 3 header))
+         (file buffer-file-name)
+         (vlf-occur-buffer
+          (generate-new-buffer (concat "*VLF-occur "
+                                       (file-name-nondirectory file)
+                                       "*"))))
+    (switch-to-buffer vlf-occur-buffer)
+    (setq buffer-file-name file
+          buffer-undo-list t)
+    (goto-char (point-min))
+    (let ((match-count 0)
+          (form 0))
+      (while (setq form (ignore-errors (read vlf-occur-data-buffer)))
+        (goto-char (point-max))
+        (insert "\n:")
+        (let* ((overlay-pos (1- (point)))
+               (overlay (make-overlay overlay-pos (1+ overlay-pos)))
+               (line (number-to-string (nth 0 form)))
+               (pos (point)))
+          (overlay-put overlay 'before-string
+                       (propertize line 'face 'shadow))
+          (insert (propertize (nth 4 form) 'chunk-start (nth 1 form)
+                              'chunk-end (nth 2 form)
+                              'mouse-face '(highlight)
+                              'line-pos (nth 3 form)
+                              'help-echo (concat "Move to line "
+                                                 line)))
+          (goto-char pos)
+          (while (re-search-forward regexp nil t)
+            (add-text-properties
+             (match-beginning 0) (match-end 0)
+             (list 'face 'match 'help-echo
+                   (format "Move to match %d"
+                           (setq match-count (1+ match-count))))))))
+      (kill-buffer vlf-occur-data-buffer)
+      (goto-char (point-min))
+      (insert (propertize
+               (format "%d matches from %d lines for \"%s\" in file: %s"
+                       match-count all-lines regexp vlf-file)
+               'face 'underline)))
+    (set-buffer-modified-p nil)
+    (vlf-occur-mode)
+    (setq vlf-occur-vlf-file vlf-file
+          vlf-occur-regexp regexp
+          vlf-occur-hexl (nth 2 header)
+          vlf-occur-lines all-lines)))
 
 (provide 'vlf-occur)
 

vlf-search.el

@@ -29,12 +29,22 @@
 
 (require 'vlf)
 
-(defun vlf-re-search (regexp count backward batch-step)
+(defun vlf-re-search (regexp count backward batch-step
+                             &optional reporter time)
   "Search for REGEXP COUNT number of times forward or BACKWARD.
-BATCH-STEP is amount of overlap between successive chunks."
+BATCH-STEP is amount of overlap between successive chunks.
+Use existing REPORTER and start TIME if given.
+Return t if search has been at least partially successful."
   (if (<= count 0)
       (error "Count must be positive"))
   (run-hook-with-args 'vlf-before-batch-functions 'search)
+  (or reporter (setq reporter (make-progress-reporter
+                               (concat "Searching for " regexp "...")
+                               (if backward
+                                   (- vlf-file-size vlf-end-pos)
+                                 vlf-start-pos)
+                               vlf-file-size)))
+  (or time (setq time (float-time)))
   (let* ((tramp-verbose (if (boundp 'tramp-verbose)
                             (min tramp-verbose 2)))
          (case-fold-search t)
@@ -43,13 +53,10 @@ BATCH-STEP is amount of overlap between successive chunks."
          (match-start-pos (+ vlf-start-pos (position-bytes (point))))
          (match-end-pos match-start-pos)
          (to-find count)
-         (font-lock font-lock-mode)
-         (reporter (make-progress-reporter
-                    (concat "Searching for " regexp "...")
-                    (if backward
-                        (- vlf-file-size vlf-end-pos)
-                      vlf-start-pos)
-                    vlf-file-size)))
+         (is-hexl (derived-mode-p 'hexl-mode))
+         (tune-types (if is-hexl '(:hexl :dehexlify :insert :encode)
+                       '(:insert :encode)))
+         (font-lock font-lock-mode))
     (font-lock-mode 0)
     (vlf-with-undo-disabled
      (unwind-protect
@@ -68,20 +75,23 @@ BATCH-STEP is amount of overlap between successive chunks."
                                                 (match-end 0)))))
                        ((zerop vlf-start-pos)
                         (throw 'end-of-file nil))
-                       (t (let ((batch-move (- vlf-start-pos
+                       (t (vlf-tune-batch tune-types)
+                          (let ((batch-move (- vlf-start-pos
                                                (- vlf-batch-size
                                                   batch-step))))
                             (vlf-move-to-batch
-                             (if (< match-start-pos batch-move)
-                                 (- match-start-pos vlf-batch-size)
-                               batch-move) t))
-                          (goto-char (if (< match-start-pos
-                                            vlf-end-pos)
-                                         (or (byte-to-position
-                                              (- match-start-pos
-                                                 vlf-start-pos))
-                                             (point-max))
-                                       (point-max)))
+                             (if (or is-hexl
+                                     (<= batch-move match-start-pos))
+                                 batch-move
+                               (- match-start-pos vlf-batch-size)) t))
+                          (goto-char (if (or is-hexl
+                                             (<= vlf-end-pos
+                                                 match-start-pos))
+                                         (point-max)
+                                       (or (byte-to-position
+                                            (- match-start-pos
+                                               vlf-start-pos))
+                                           (point-max))))
                           (progress-reporter-update
                            reporter (- vlf-file-size
                                        vlf-start-pos)))))
@@ -98,50 +108,60 @@ BATCH-STEP is amount of overlap between successive chunks."
                                               (match-end 0)))))
                      ((= vlf-end-pos vlf-file-size)
                       (throw 'end-of-file nil))
-                     (t (let ((batch-move (- vlf-end-pos batch-step)))
+                     (t (vlf-tune-batch tune-types)
+                        (let ((batch-move (- vlf-end-pos batch-step)))
                           (vlf-move-to-batch
-                           (if (< batch-move match-end-pos)
-                               match-end-pos
-                             batch-move) t))
-                        (goto-char (if (< vlf-start-pos match-end-pos)
-                                       (or (byte-to-position
-                                            (- match-end-pos
-                                               vlf-start-pos))
-                                           (point-min))
-                                     (point-min)))
+                           (if (or is-hexl
+                                   (< match-end-pos batch-move))
+                               batch-move
+                             match-end-pos) t))
+                        (goto-char (if (or is-hexl
+                                           (<= match-end-pos vlf-start-pos))
+                                       (point-min)
+                                     (or (byte-to-position
+                                          (- match-end-pos
+                                             vlf-start-pos))
+                                         (point-min))))
                         (progress-reporter-update reporter
                                                   vlf-end-pos)))))
            (progress-reporter-done reporter))
        (set-buffer-modified-p nil)
+       (if is-hexl (vlf-tune-hexlify))
        (if font-lock (font-lock-mode 1))
-       (if backward
-           (vlf-goto-match match-chunk-start match-chunk-end
-                           match-end-pos match-start-pos
-                           count to-find)
-         (vlf-goto-match match-chunk-start match-chunk-end
-                         match-start-pos match-end-pos
-                         count to-find))
-       (run-hook-with-args 'vlf-after-batch-functions 'search)))))
+       (let ((result
+              (if backward
+                  (vlf-goto-match match-chunk-start match-chunk-end
+                                  match-end-pos match-start-pos
+                                  count to-find time)
+                (vlf-goto-match match-chunk-start match-chunk-end
+                                match-start-pos match-end-pos
+                                count to-find time))))
+         (run-hook-with-args 'vlf-after-batch-functions 'search)
+         result)))))
 
 (defun vlf-goto-match (match-chunk-start match-chunk-end
-                                         match-pos-start
-                                         match-pos-end
-                                         count to-find)
+                                         match-pos-start match-pos-end
+                                         count to-find time)
   "Move to MATCH-CHUNK-START MATCH-CHUNK-END surrounding\
 MATCH-POS-START and MATCH-POS-END.
 According to COUNT and left TO-FIND, show if search has been
-successful.  Return nil if nothing found."
+successful.  Use start TIME to report how much it took.
+Return nil if nothing found."
   (if (= count to-find)
       (progn (vlf-move-to-chunk match-chunk-start match-chunk-end)
              (goto-char (or (byte-to-position (- match-pos-start
                                                  vlf-start-pos))
                             (point-max)))
-             (message "Not found")
+             (message "Not found (%f secs)" (- (float-time) time))
              nil)
     (let ((success (zerop to-find)))
       (if success
           (vlf-update-buffer-name)
         (vlf-move-to-chunk match-chunk-start match-chunk-end))
+      (setq vlf-batch-size (vlf-tune-optimal-load
+                            (if (derived-mode-p 'hexl-mode)
+                                '(:hexl :dehexlify :insert :encode)
+                              '(:insert :encode))))
       (let* ((match-end (or (byte-to-position (- match-pos-end
                                                  vlf-start-pos))
                             (point-max)))
@@ -150,10 +170,11 @@ successful.  Return nil if nothing found."
                                         vlf-start-pos))
                                     match-end)))
         (overlay-put overlay 'face 'match)
-        (unless success
+        (if success
+            (message "Match found (%f secs)" (- (float-time) time))
           (goto-char match-end)
-          (message "Moved to the %d match which is last"
-                   (- count to-find)))
+          (message "Moved to the %d match which is last (%f secs)"
+                   (- count to-find) (- (float-time) time)))
         (unwind-protect (sit-for 3)
           (delete-overlay overlay))
         t))))
@@ -166,7 +187,9 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (vlf-re-search regexp count nil (/ vlf-batch-size 8)))
+  (let ((batch-size vlf-batch-size))
+    (or (vlf-re-search regexp count nil (min 1024 (/ vlf-batch-size 8)))
+        (setq vlf-batch-size batch-size))))
 
 (defun vlf-re-search-backward (regexp count)
   "Search backward for REGEXP prefix COUNT number of times.
@@ -176,7 +199,9 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                                       (if regexp-history
                                           (car regexp-history)))
                          (or current-prefix-arg 1))))
-  (vlf-re-search regexp count t (/ vlf-batch-size 8)))
+  (let ((batch-size vlf-batch-size))
+    (or (vlf-re-search regexp count t (min 1024 (/ vlf-batch-size 8)))
+        (setq vlf-batch-size batch-size))))
 
 (defun vlf-goto-line (n)
   "Go to line N.  If N is negative, count from the end of file."
@@ -188,10 +213,14 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                            (min tramp-verbose 2)))
         (start-pos vlf-start-pos)
         (end-pos vlf-end-pos)
+        (batch-size vlf-batch-size)
         (pos (point))
+        (is-hexl (derived-mode-p 'hexl-mode))
         (font-lock font-lock-mode)
+        (time (float-time))
         (success nil))
     (font-lock-mode 0)
+    (vlf-tune-batch '(:raw))
     (unwind-protect
         (if (< 0 n)
             (let ((start 0)
@@ -203,23 +232,28 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                   (inhibit-read-only t))
               (setq n (1- n))
               (vlf-with-undo-disabled
-               (while (and (< (- end start) n)
-                           (< n (- vlf-file-size start)))
-                 (erase-buffer)
-                 (insert-file-contents-literally buffer-file-name
-                                                 nil start end)
-                 (goto-char (point-min))
-                 (while (re-search-forward "[\n\C-m]" nil t)
-                   (setq n (1- n)))
-                 (vlf-verify-size)
-                 (setq start end
-                       end (min vlf-file-size
-                                (+ start vlf-batch-size)))
-                 (progress-reporter-update reporter start))
+               (or is-hexl
+                   (while (and (< (- end start) n)
+                               (< n (- vlf-file-size start)))
+                     (erase-buffer)
+                     (vlf-tune-insert-file-contents-literally start end)
+                     (goto-char (point-min))
+                     (while (re-search-forward "[\n\C-m]" nil t)
+                       (setq n (1- n)))
+                     (vlf-verify-size)
+                     (vlf-tune-batch '(:raw))
+                     (setq start end
+                           end (min vlf-file-size
+                                    (+ start vlf-batch-size)))
+                     (progress-reporter-update reporter start)))
                (when (< n (- vlf-file-size end))
-                 (vlf-move-to-chunk-2 start end)
+                 (vlf-tune-batch (if is-hexl
+                                     '(:hexl :dehexlify :insert :encode)
+                                   '(:insert :encode)))
+                 (vlf-move-to-chunk-2 start (+ start vlf-batch-size))
                  (goto-char (point-min))
-                 (setq success (vlf-re-search "[\n\C-m]" n nil 0)))))
+                 (setq success (vlf-re-search "[\n\C-m]" n nil 0
+                                              reporter time)))))
           (let ((start (max 0 (- vlf-file-size vlf-batch-size)))
                 (end vlf-file-size)
                 (reporter (make-progress-reporter
@@ -229,27 +263,33 @@ Search is performed chunk by chunk in `vlf-batch-size' memory."
                 (inhibit-read-only t))
             (setq n (- n))
             (vlf-with-undo-disabled
-             (while (and (< (- end start) n) (< n end))
-               (erase-buffer)
-               (insert-file-contents-literally buffer-file-name nil
-                                               start end)
-               (goto-char (point-max))
-               (while (re-search-backward "[\n\C-m]" nil t)
-                 (setq n (1- n)))
-               (setq end start
-                     start (max 0 (- end vlf-batch-size)))
-               (progress-reporter-update reporter
-                                         (- vlf-file-size end)))
+             (or is-hexl
+                 (while (and (< (- end start) n) (< n end))
+                   (erase-buffer)
+                   (vlf-tune-insert-file-contents-literally start end)
+                   (goto-char (point-max))
+                   (while (re-search-backward "[\n\C-m]" nil t)
+                     (setq n (1- n)))
+                   (vlf-tune-batch '(:raw))
+                   (setq end start
+                         start (max 0 (- end vlf-batch-size)))
+                   (progress-reporter-update reporter
+                                             (- vlf-file-size end))))
              (when (< n end)
-               (vlf-move-to-chunk-2 start end)
+               (vlf-tune-batch (if is-hexl
+                                   '(:hexl :dehexlify :insert :encode)
+                                 '(:insert :encode)))
+               (vlf-move-to-chunk-2 (- end vlf-batch-size) end)
                (goto-char (point-max))
-               (setq success (vlf-re-search "[\n\C-m]" n t 0))))))
+               (setq success (vlf-re-search "[\n\C-m]" n t 0
+                                            reporter time))))))
       (if font-lock (font-lock-mode 1))
       (unless success
         (vlf-with-undo-disabled
          (vlf-move-to-chunk-2 start-pos end-pos))
         (vlf-update-buffer-name)
         (goto-char pos)
+        (setq vlf-batch-size batch-size)
         (message "Unable to find line"))
       (run-hook-with-args 'vlf-after-batch-functions 'goto-line))))
 

vlf-tune.el

@@ -0,0 +1,423 @@
+;;; vlf-tune.el --- VLF tuning operations  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2014 Free Software Foundation, Inc.
+
+;; Keywords: large files, batch size, performance
+;; Author: Andrey Kotlarski <m00naticus@gmail.com>
+;; URL: https://github.com/m00natic/vlfi
+
+;; This file is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+
+;; This file is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs; see the file COPYING.  If not, write to
+;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+;; Boston, MA 02111-1307, USA.
+
+;;; Commentary:
+;; This package provides wrappers for basic chunk operations that add
+;; profiling and automatic tuning of `vlf-batch-size'.
+
+;;; Code:
+
+(defgroup vlf nil "View Large Files in Emacs."
+  :prefix "vlf-" :group 'files)
+
+(defcustom vlf-batch-size 1000000
+  "Defines how large each batch of file data initially is (in bytes)."
+  :group 'vlf :type 'integer)
+(put 'vlf-batch-size 'permanent-local t)
+
+(defcustom vlf-tune-enabled t
+  "Whether to allow automatic change of batch size.
+If nil, completely disable.  If `stats', maintain measure statistics,
+but don't change batch size.  If t, measure and change."
+  :group 'vlf :type '(choice (const :tag "Enabled" t)
+                             (const :tag "Just statistics" stats)
+                             (const :tag "Disabled" nil)))
+
+(defvar vlf-file-size 0 "Total size in bytes of presented file.")
+(make-variable-buffer-local 'vlf-file-size)
+(put 'vlf-file-size 'permanent-local t)
+
+(defun vlf-tune-ram-size ()
+  "Try to determine RAM size in bytes."
+  (if (executable-find "free")
+      (let* ((free (shell-command-to-string "free"))
+             (match-from (string-match "[[:digit:]]+" free)))
+        (if match-from
+            (* 1000 (string-to-number (substring free match-from
+                                                 (match-end 0))))))))
+
+(defcustom vlf-tune-max (let ((ram-size (vlf-tune-ram-size)))
+                          (if ram-size
+                              (/ ram-size 20)
+                            large-file-warning-threshold))
+  "Maximum batch size in bytes when auto tuning."
+  :group 'vlf :type 'integer)
+
+(defcustom vlf-tune-step (/ vlf-tune-max 1000)
+  "Step used for tuning in bytes."
+  :group 'vlf :type 'integer)
+
+(defcustom vlf-tune-load-time 1.0
+  "How many seconds should batch take to load for best user experience."
+  :group 'vlf :type 'float)
+
+(defvar vlf-tune-insert-bps nil
+  "Vector of bytes per second insert measurements.")
+(make-variable-buffer-local 'vlf-tune-insert-bps)
+(put 'vlf-tune-insert-bps 'permanent-local t)
+
+(defvar vlf-tune-insert-raw-bps nil
+  "Vector of bytes per second non-decode insert measurements.")
+(make-variable-buffer-local 'vlf-tune-insert-raw-bps)
+(put 'vlf-tune-insert-raw-bps 'permanent-local t)
+
+(defvar vlf-tune-encode-bps nil
+  "Vector of bytes per second encode measurements.")
+(make-variable-buffer-local 'vlf-tune-encode-bps)
+(put 'vlf-tune-encode-bps 'permanent-local t)
+
+(defvar vlf-tune-write-bps nil
+  "Vector of bytes per second write measurements.")
+(make-variable-buffer-local 'vlf-tune-write-bps)
+(put 'vlf-tune-write-bps 'permanent-local t)
+
+(defvar vlf-tune-hexl-bps nil
+  "Vector of bytes per second hexlify measurements.")
+(make-variable-buffer-local 'vlf-tune-hexl-bps)
+(put 'vlf-tune-hexl-bps 'permanent-local t)
+
+(defvar vlf-tune-dehexlify-bps nil
+  "Vector of bytes per second dehexlify measurements.")
+(make-variable-buffer-local 'vlf-tune-dehexlify-bps)
+(put 'vlf-tune-dehexlify-bps 'permanent-local t)
+
+(defun vlf-tune-closest-index (size)
+  "Get closest measurement index corresponding to SIZE."
+  (let ((step (float vlf-tune-step)))
+    (max 0 (1- (min (round size step) (round vlf-tune-max step))))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; profiling
+
+(defun vlf-tune-initialize-measurement ()
+  "Initialize measurement vector."
+  (make-local-variable 'vlf-tune-max)
+  (make-local-variable 'vlf-tune-step)
+  (make-vector (/ vlf-tune-max vlf-tune-step) nil))
+
+(defmacro vlf-tune-add-measurement (vec size time)
+  "Add at an appropriate position in VEC new SIZE TIME measurement.
+VEC is a vector of (mean time . count) elements ordered by size."
+  `(when (and vlf-tune-enabled (not (zerop ,size)))
+     (or ,vec (setq ,vec (vlf-tune-initialize-measurement)))
+     (let* ((idx (vlf-tune-closest-index ,size))
+            (existing (aref ,vec idx)))
+       (aset ,vec idx (if (consp existing)
+                          (let ((count (1+ (cdr existing)))) ;recalculate mean
+                            (cons (/ (+ (* (1- count) (car existing))
+                                        (/ ,size ,time))
+                                     count)
+                                  count))
+                        (cons (/ ,size ,time) 1))))))
+
+(defmacro vlf-time (&rest body)
+  "Get timing consed with result of BODY execution."
+  `(if vlf-tune-enabled
+       (let* ((time (float-time))
+              (result (progn ,@body)))
+         (cons (- (float-time) time) result))
+     (let ((result (progn ,@body)))
+       (cons nil result))))
+
+(defun vlf-tune-insert-file-contents (start end)
+  "Extract decoded file bytes START to END and save time it takes."
+  (let ((result (vlf-time (insert-file-contents buffer-file-name
+                                                nil start end))))
+    (vlf-tune-add-measurement vlf-tune-insert-bps
+                              (- end start) (car result))
+    (cdr result)))
+
+(defun vlf-tune-insert-file-contents-literally (start end)
+  "Insert raw file bytes START to END and save time it takes."
+  (let ((result (vlf-time (insert-file-contents-literally
+                           buffer-file-name nil start end))))
+    (vlf-tune-add-measurement vlf-tune-insert-raw-bps
+                              (- end start) (car result))
+    (cdr result)))
+
+(defun vlf-tune-encode-length (start end)
+  "Get length of encoded region START to END and save time it takes."
+  (let ((result (vlf-time (length (encode-coding-region
+                                   start end
+                                   buffer-file-coding-system t)))))
+    (vlf-tune-add-measurement vlf-tune-encode-bps
+                              (cdr result) (car result))
+    (cdr result)))
+
+(defun vlf-tune-write (start end append visit size)
+  "Save buffer and save time it takes.
+START, END, APPEND, VISIT have same meaning as in `write-region'.
+SIZE is number of bytes that are saved."
+  (let ((time (car (vlf-time (write-region start end buffer-file-name
+                                           append visit)))))
+    (vlf-tune-add-measurement vlf-tune-write-bps size time)))
+
+(defun vlf-tune-hexlify ()
+  "Activate `hexl-mode' and save time it takes."
+  (or (derived-mode-p 'hexl-mode)
+      (let ((time (car (vlf-time (hexl-mode)))))
+        (vlf-tune-add-measurement vlf-tune-hexl-bps
+                                  hexl-max-address time))))
+
+(defun vlf-tune-dehexlify ()
+  "Exit `hexl-mode' and save time it takes."
+  (if (derived-mode-p 'hexl-mode)
+      (let ((time (car (vlf-time (hexl-mode-exit)))))
+        (vlf-tune-add-measurement vlf-tune-dehexlify-bps
+                                  hexl-max-address time))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; tuning
+
+(defun vlf-tune-approximate-nearby (vec index)
+  "VEC has value for INDEX, approximate to closest available."
+  (let ((val 0)
+        (left-idx (1- index))
+        (right-idx (1+ index))
+        (min-idx (max 0 (- index 5)))
+        (max-idx (min (+ index 6)
+                      (1- (/ (min vlf-tune-max
+                                  (/ (1+ vlf-file-size) 2))
+                             vlf-tune-step)))))
+    (while (and (zerop val) (or (<= min-idx left-idx)
+                                (< right-idx max-idx)))
+      (if (<= min-idx left-idx)
+          (let ((left (aref vec left-idx)))
+            (cond ((consp left) (setq val (car left)))
+                  ((numberp left) (setq val left)))))
+      (if (< right-idx max-idx)
+          (let ((right (aref vec right-idx)))
+            (if (consp right)
+                (setq right (car right)))
+            (and (numberp right) (not (zerop right))
+                 (setq val (if (zerop val)
+                               right
+                             (/ (+ val right) 2))))))
+      (setq left-idx (1- left-idx)
+            right-idx (1+ right-idx)))
+    val))
+
+(defmacro vlf-tune-get-value (vec index &optional dont-approximate)
+  "Get value from VEC for INDEX.
+If missing, approximate from nearby measurement,
+unless DONT-APPROXIMATE is t."
+  `(if ,vec
+       (let ((val (aref ,vec ,index)))
+         (cond ((consp val) (car val))
+               ((null val)
+                ,(if dont-approximate
+                     `(aset ,vec ,index 0)
+                   `(vlf-tune-approximate-nearby ,vec ,index)))
+               ((zerop val) ;index has been tried before, yet still no value
+                ,(if dont-approximate
+                     `(aset ,vec ,index
+                            (vlf-tune-approximate-nearby ,vec ,index))
+                   `(vlf-tune-approximate-nearby ,vec ,index)))
+               (t val)))))
+
+(defmacro vlf-tune-get-vector (key)
+  "Get vlf-tune vector corresponding to KEY."
+  `(cond ((eq ,key :insert) vlf-tune-insert-bps)
+         ((eq ,key :raw) vlf-tune-insert-raw-bps)
+         ((eq ,key :encode) vlf-tune-encode-bps)
+         ((eq ,key :write) vlf-tune-write-bps)
+         ((eq ,key :hexl) vlf-tune-hexl-bps)
+         ((eq ,key :dehexlify) vlf-tune-dehexlify-bps)))
+
+(defun vlf-tune-assess (type coef index &optional approximate)
+  "Get measurement value according to TYPE, COEF and INDEX.
+If APPROXIMATE is t, do approximation for missing values."
+  (* coef (or (if approximate
+                  (vlf-tune-get-value (vlf-tune-get-vector type)
+                                      index)
+                (vlf-tune-get-value (vlf-tune-get-vector type)
+                                    index t))
+              0)))
+
+(defun vlf-tune-score (types index &optional approximate time-max)
+  "Calculate cumulative speed over TYPES for INDEX.
+If APPROXIMATE is t, do approximation for missing values.
+If TIME-MAX is non nil, return cumulative time instead of speed.
+If it is number, stop as soon as cumulative time gets equal or above."
+  (catch 'result
+    (let ((time 0)
+          (size (* (1+ index) vlf-tune-step))
+          (cut-time (numberp time-max)))
+      (dolist (el types (if time-max time
+                          (/ size time)))
+        (let ((bps (if (consp el)
+                       (vlf-tune-assess (car el) (cadr el) index
+                                        approximate)
+                     (vlf-tune-assess el 1 index approximate))))
+          (if (zerop bps)
+              (throw 'result nil)
+            (setq time (+ time (/ size bps)))
+            (and cut-time (<= time-max time)
+                 (throw 'result nil))))))))
+
+(defun vlf-tune-conservative (types &optional index)
+  "Adjust `vlf-batch-size' to best nearby value over TYPES.
+INDEX if given, specifies search independent of current batch size."
+  (if (eq vlf-tune-enabled t)
+      (let* ((half-max (/ (1+ vlf-file-size) 2))
+             (idx (or index (vlf-tune-closest-index vlf-batch-size)))
+             (curr (if (< half-max (* idx vlf-tune-step)) t
+                     (vlf-tune-score types idx))))
+        (if curr
+            (let ((prev (if (zerop idx) t
+                          (vlf-tune-score types (1- idx)))))
+              (if prev
+                  (let ((next (if (or (eq curr t)
+                                      (< half-max (* (1+ idx)
+                                                     vlf-tune-step)))
+                                  t
+                                (vlf-tune-score types (1+ idx)))))
+                    (cond ((null next)
+                           (setq vlf-batch-size (* (+ 2 idx)
+                                                   vlf-tune-step)))
+                          ((eq curr t)
+                           (or (eq prev t)
+                               (setq vlf-batch-size
+                                     (* idx vlf-tune-step))))
+                          (t (let ((best-idx idx))
+                               (and (numberp prev) (< curr prev)
+                                    (setq curr prev
+                                          best-idx (1- idx)))
+                               (and (numberp next) (< curr next)
+                                    (setq best-idx (1+ idx)))
+                               (setq vlf-batch-size
+                                     (* (1+ best-idx)
+                                        vlf-tune-step))))))
+                (setq vlf-batch-size (* idx vlf-tune-step))))
+          (setq vlf-batch-size (* (1+ idx) vlf-tune-step))))))
+
+(defun vlf-tune-binary (types min max)
+  "Adjust `vlf-batch-size' to optimal value using binary search, \
+optimizing over TYPES.
+MIN and MAX specify interval of indexes to search."
+  (let ((sum (+ min max)))
+    (if (< (- max min) 3)
+        (vlf-tune-conservative types (/ sum 2))
+      (let* ((left-idx (round (+ sum (* 2 min)) 4))
+             (left (vlf-tune-score types left-idx)))
+        (if left
+            (let* ((right-idx (round (+ sum (* 2 max)) 4))
+                   (right (vlf-tune-score types right-idx)))
+              (cond ((null right)
+                     (setq vlf-batch-size (* (1+ right-idx)
+                                             vlf-tune-step)))
+                    ((< left right)
+                     (vlf-tune-binary types (/ (1+ sum) 2) max))
+                    (t (vlf-tune-binary types min (/ sum 2)))))
+          (setq vlf-batch-size (* (1+ left-idx) vlf-tune-step)))))))
+
+(defun vlf-tune-linear (types max-idx)
+  "Adjust `vlf-batch-size' to optimal value using linear search, \
+optimizing over TYPES up to MAX-IDX."
+  (let ((best-idx 0)
+        (best-bps 0)
+        (idx 0)
+        (none-missing t))
+    (while (and none-missing (< idx max-idx))
+      (let ((bps (vlf-tune-score types idx)))
+        (cond ((null bps)
+               (setq vlf-batch-size (* (1+ idx) vlf-tune-step)
+                     none-missing nil))
+              ((< best-bps bps) (setq best-idx idx
+                                      best-bps bps))))
+      (setq idx (1+ idx)))
+    (or (not none-missing)
+        (setq vlf-batch-size (* (1+ best-idx) vlf-tune-step)))))
+
+(defun vlf-tune-batch (types &optional linear)
+  "Adjust `vlf-batch-size' to optimal value optimizing on TYPES.
+TYPES is alist of elements that may be of form (type coef) or
+non list values in which case coeficient is assumed 1.
+Types can be :insert, :raw, :encode, :write, :hexl or :dehexlify.
+If LINEAR is non nil, use brute-force.  In case requested measurement
+is missing, stop search and set `vlf-batch-size' to this value.
+Suitable for multiple batch operations."
+  (if (eq vlf-tune-enabled t)
+      (let ((max-idx (1- (/ (min vlf-tune-max
+                                 (/ (1+ vlf-file-size) 2))
+                            vlf-tune-step))))
+        (cond (linear (vlf-tune-linear types max-idx))
+              ((file-remote-p buffer-file-name)
+               (vlf-tune-conservative types))
+              ((<= 1 max-idx)
+               (if (< max-idx 3)
+                   (vlf-tune-conservative types (/ max-idx 2))
+                 (vlf-tune-binary types 0 max-idx)))))))
+
+(defun vlf-tune-optimal-load (types &optional min-idx max-idx)
+  "Get best batch size according to existing measurements over TYPES.
+Best considered where primitive operations total is closest to
+`vlf-tune-load-time'.  If MIN-IDX and MAX-IDX are given,
+confine search to this region."
+  (if vlf-tune-enabled
+      (progn
+        (setq max-idx (min (or max-idx vlf-tune-max)
+                           (1- (/ (min vlf-tune-max
+                                       (/ (1+ vlf-file-size) 2))
+                                  vlf-tune-step))))
+        (let* ((idx (max 0 (or min-idx 0)))
+               (best-idx idx)
+               (best-time-diff vlf-tune-load-time)
+               (all-less t)
+               (all-more t))
+          (while (and (not (zerop best-time-diff)) (< idx max-idx))
+            (let ((time-diff (vlf-tune-score types idx t
+                                             (+ vlf-tune-load-time
+                                                best-time-diff))))
+              (if time-diff
+                  (progn
+                    (setq time-diff (if (< vlf-tune-load-time time-diff)
+                                        (progn (setq all-less nil)
+                                               (- time-diff
+                                                  vlf-tune-load-time))
+                                      (setq all-more nil)
+                                      (- vlf-tune-load-time time-diff)))
+                    (if (< time-diff best-time-diff)
+                        (setq best-idx idx
+                              best-time-diff time-diff)))
+                (setq all-less nil)))
+            (setq idx (1+ idx)))
+          (* vlf-tune-step (1+ (cond ((or (zerop best-time-diff)
+                                          (eq all-less all-more))
+                                      best-idx)
+                                     (all-less max-idx)
+                                     (t min-idx))))))
+    vlf-batch-size))
+
+(defun vlf-tune-load (types &optional region)
+  "Adjust `vlf-batch-size' slightly to better load time.
+Optimize on TYPES on the nearby REGION.  Use 2 if REGION is nil."
+  (when (eq vlf-tune-enabled t)
+    (or region (setq region 2))
+    (let ((idx (vlf-tune-closest-index vlf-batch-size)))
+      (setq vlf-batch-size (vlf-tune-optimal-load types (- idx region)
+                                                  (+ idx 1 region))))))
+
+(provide 'vlf-tune)
+
+;;; vlf-tune.el ends here

vlf-write.el

@@ -37,43 +37,56 @@ If changing size of chunk, shift remaining file content."
              (or (verify-visited-file-modtime (current-buffer))
                  (y-or-n-p "File has changed since visited or saved.\
   Save anyway? ")))
+    (widen)
     (run-hook-with-args 'vlf-before-batch-functions 'write)
-    (if (zerop vlf-file-size)           ;new file
-        (progn (write-region nil nil buffer-file-name vlf-start-pos t)
-               (setq vlf-file-size (vlf-get-file-size
-                                    buffer-file-truename)
-                     vlf-end-pos vlf-file-size)
-               (vlf-update-buffer-name))
-      (widen)
-      (let* ((region-length (length (encode-coding-region
-                                     (point-min) (point-max)
-                                     buffer-file-coding-system t)))
-             (size-change (- vlf-end-pos vlf-start-pos
-                             region-length)))
-        (if (zerop size-change)
-            (write-region nil nil buffer-file-name vlf-start-pos t)
-          (let ((tramp-verbose (if (boundp 'tramp-verbose)
-                                   (min tramp-verbose 2)))
-                (pos (point))
-                (font-lock font-lock-mode))
-            (font-lock-mode 0)
-            (if (< 0 size-change)
-                (vlf-file-shift-back size-change)
-              (vlf-file-shift-forward (- size-change)))
-            (if font-lock (font-lock-mode 1))
-            (vlf-move-to-chunk-2 vlf-start-pos
-                                 (if (< (- vlf-end-pos vlf-start-pos)
-                                        vlf-batch-size)
-                                     (+ vlf-start-pos vlf-batch-size)
-                                   vlf-end-pos))
-            (vlf-update-buffer-name)
-            (goto-char pos)))))
+    (let ((hexl (derived-mode-p 'hexl-mode)))
+      (when hexl
+        (if (consp buffer-undo-list)
+            (setq buffer-undo-list nil))
+        (vlf-tune-dehexlify))
+      (if (zerop vlf-file-size)           ;new file
+          (progn (vlf-tune-write nil nil vlf-start-pos t
+                                 (vlf-tune-encode-length (point-min)
+                                                         (point-max)))
+                 (setq vlf-file-size (vlf-get-file-size
+                                      buffer-file-truename)
+                       vlf-end-pos vlf-file-size)
+                 (vlf-update-buffer-name))
+        (let* ((region-length (vlf-tune-encode-length (point-min)
+                                                      (point-max)))
+               (size-change (- vlf-end-pos vlf-start-pos
+                               region-length)))
+          (if (zerop size-change)
+              (vlf-tune-write nil nil vlf-start-pos t
+                              (- vlf-end-pos vlf-start-pos))
+            (let ((tramp-verbose (if (boundp 'tramp-verbose)
+                                     (min tramp-verbose 2)))
+                  (pos (point))
+                  (font-lock font-lock-mode))
+              (font-lock-mode 0)
+              (let ((batch-size vlf-batch-size)
+                    (time (float-time)))
+                (if (< 0 size-change)
+                    (vlf-file-shift-back size-change region-length)
+                  (vlf-file-shift-forward (- size-change) region-length))
+                (if font-lock (font-lock-mode 1))
+                (setq vlf-batch-size batch-size)
+                (vlf-move-to-chunk-2 vlf-start-pos
+                                     (if (< (- vlf-end-pos vlf-start-pos)
+                                            vlf-batch-size)
+                                         (+ vlf-start-pos vlf-batch-size)
+                                       vlf-end-pos))
+                (vlf-update-buffer-name)
+                (goto-char pos)
+                (message "Save took %f seconds" (- (float-time) time)))))))
+      (if hexl (vlf-tune-hexlify)))
     (run-hook-with-args 'vlf-after-batch-functions 'write))
   t)
 
-(defun vlf-file-shift-back (size-change)
-  "Shift file contents SIZE-CHANGE bytes back."
-  (write-region nil nil buffer-file-name vlf-start-pos t)
+(defun vlf-file-shift-back (size-change write-size)
+  "Shift file contents SIZE-CHANGE bytes back.
+WRITE-SIZE is byte length of saved chunk."
+  (vlf-tune-write nil nil vlf-start-pos t write-size)
   (let ((read-start-pos vlf-end-pos)
         (coding-system-for-write 'no-conversion)
         (reporter (make-progress-reporter "Adjusting file content..."
@@ -88,8 +101,8 @@ If changing size of chunk, shift remaining file content."
      (erase-buffer)
      (vlf-verify-size t)
      (insert-char 32 size-change))
-    (write-region nil nil buffer-file-name (- vlf-file-size
-                                              size-change) t)
+    (vlf-tune-write nil nil (- vlf-file-size size-change)
+                    t size-change)
     (progress-reporter-done reporter)))
 
 (defun vlf-shift-batch (read-pos write-pos)
@@ -97,37 +110,48 @@ If changing size of chunk, shift remaining file content."
 back at WRITE-POS.  Return nil if EOF is reached, t otherwise."
   (erase-buffer)
   (vlf-verify-size t)
-  (let ((read-end (+ read-pos vlf-batch-size)))
-    (insert-file-contents-literally buffer-file-name nil
-                                    read-pos
-                                    (min vlf-file-size read-end))
-    (write-region nil nil buffer-file-name write-pos 0)
+  (vlf-tune-batch '(:raw :write))
+  (let ((read-end (min (+ read-pos vlf-batch-size) vlf-file-size)))
+    (vlf-tune-insert-file-contents-literally read-pos read-end)
+    (vlf-tune-write nil nil write-pos 0 (- read-end read-pos))
     (< read-end vlf-file-size)))
 
-(defun vlf-file-shift-forward (size-change)
+(defun vlf-file-shift-forward (size-change write-size)
   "Shift file contents SIZE-CHANGE bytes forward.
+WRITE-SIZE is byte length of saved chunk.
 Done by saving content up front and then writing previous batch."
-  (let ((read-size (max (/ vlf-batch-size 2) size-change))
+  (vlf-tune-batch '(:raw :write))
+  (let ((read-size (max vlf-batch-size size-change))
         (read-pos vlf-end-pos)
         (write-pos vlf-start-pos)
         (reporter (make-progress-reporter "Adjusting file content..."
                                           vlf-start-pos
                                           vlf-file-size)))
     (vlf-with-undo-disabled
-     (when (vlf-shift-batches read-size read-pos write-pos t)
+     (when (vlf-shift-batches read-size read-pos write-pos
+                              write-size t)
+       (vlf-tune-batch '(:raw :write))
        (setq write-pos (+ read-pos size-change)
-             read-pos (+ read-pos read-size))
+             read-pos (+ read-pos read-size)
+             write-size read-size
+             read-size (max vlf-batch-size size-change))
        (progress-reporter-update reporter write-pos)
        (let ((coding-system-for-write 'no-conversion))
-         (while (vlf-shift-batches read-size read-pos write-pos nil)
+         (while (vlf-shift-batches read-size read-pos write-pos
+                                   write-size nil)
+           (vlf-tune-batch '(:raw :write))
            (setq write-pos (+ read-pos size-change)
-                 read-pos (+ read-pos read-size))
+                 read-pos (+ read-pos read-size)
+                 write-size read-size
+                 read-size (max vlf-batch-size size-change))
            (progress-reporter-update reporter write-pos)))))
     (progress-reporter-done reporter)))
 
-(defun vlf-shift-batches (read-size read-pos write-pos hide-read)
+(defun vlf-shift-batches (read-size read-pos write-pos write-size
+                                    hide-read)
   "Append READ-SIZE bytes of file starting at READ-POS.
 Then write initial buffer content to file at WRITE-POS.
+WRITE-SIZE is byte length of saved chunk.
 If HIDE-READ is non nil, temporarily hide literal read content.
 Return nil if EOF is reached, t otherwise."
   (vlf-verify-size t)
@@ -136,14 +160,13 @@ Return nil if EOF is reached, t otherwise."
         (end-write-pos (point-max)))
     (when read-more
       (goto-char end-write-pos)
-      (insert-file-contents-literally buffer-file-name nil read-pos
-                                      (min vlf-file-size
-                                           (+ read-pos read-size))))
+      (vlf-tune-insert-file-contents-literally
+       read-pos (min vlf-file-size (+ read-pos read-size))))
     ;; write
     (if hide-read ; hide literal region if user has to choose encoding
         (narrow-to-region start-write-pos end-write-pos))
-    (write-region start-write-pos end-write-pos
-                  buffer-file-name write-pos 0)
+    (vlf-tune-write start-write-pos end-write-pos write-pos
+                    (or (not read-more) 0) write-size)
     (delete-region start-write-pos end-write-pos)
     (if hide-read (widen))
     read-more))

vlf.el

@@ -2,7 +2,7 @@
 
 ;; Copyright (C) 2006, 2012-2014 Free Software Foundation, Inc.
 
-;; Version: 1.5
+;; Version: 1.6
 ;; Keywords: large files, utilities
 ;; Maintainer: Andrey Kotlarski <m00naticus@gmail.com>
 ;; Authors: 2006 Mathias Dahl <mathias.dahl@gmail.com>
@@ -39,8 +39,7 @@
 
 ;;; Code:
 
-(defgroup vlf nil "View Large Files in Emacs."
-  :prefix "vlf-" :group 'files)
+(require 'vlf-base)
 
 (defcustom vlf-before-batch-functions nil
   "Hook that runs before multiple batch operations.
@@ -54,8 +53,6 @@ One argument is supplied that specifies current action.  Possible
 values are: `write', `ediff', `occur', `search', `goto-line'."
   :group 'vlf :type 'hook)
 
-(require 'vlf-base)
-
 (autoload 'vlf-write "vlf-write" "Write current chunk to file." t)
 (autoload 'vlf-re-search-forward "vlf-search"
   "Search forward for REGEXP prefix COUNT number of times." t)
@@ -145,17 +142,22 @@ values are: `write', `ediff', `occur', `search', `goto-line'."
   (setq vlf-mode t))
 
 ;;;###autoload
-(defun vlf (file)
-  "View Large FILE in batches.
+(defun vlf (file &optional minimal)
+  "View Large FILE in batches.  When MINIMAL load just a few bytes.
 You can customize number of bytes displayed by customizing
 `vlf-batch-size'.
 Return newly created buffer."
-  (interactive "fFile to open: ")
+  (interactive (list (read-file-name "File to open: ") nil))
   (let ((vlf-buffer (generate-new-buffer "*vlf*")))
     (set-buffer vlf-buffer)
     (set-visited-file-name file)
     (set-buffer-modified-p nil)
+    (if (or minimal (file-remote-p file))
+        (set (make-local-variable 'vlf-batch-size) 1024))
     (vlf-mode 1)
+    (when minimal                 ;restore batch size to default value
+      (kill-local-variable 'vlf-batch-size)
+      (make-local-variable 'vlf-batch-size))
     (switch-to-buffer vlf-buffer)
     vlf-buffer))
 
@@ -167,6 +169,9 @@ When prefix argument is negative
  append next APPEND number of batches to the existing buffer."
   (interactive "p")
   (vlf-verify-size)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :dehexlify :insert :encode)
+                   '(:insert :encode)))
   (let* ((end (min (+ vlf-end-pos (* vlf-batch-size (abs append)))
                    vlf-file-size))
          (start (if (< append 0)
@@ -183,6 +188,9 @@ When prefix argument is negative
   (interactive "p")
   (if (zerop vlf-start-pos)
       (error "Already at BOF"))
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :dehexlify :insert :encode)
+                   '(:insert :encode)))
   (let* ((start (max 0 (- vlf-start-pos (* vlf-batch-size (abs prepend)))))
          (end (if (< prepend 0)
                   vlf-end-pos
@@ -206,24 +214,8 @@ When prefix argument is negative
              (goto-char (point-max)))
     ad-do-it))
 
-;; hexl mode integration
-(defun vlf-hexl-before (&optional operation)
-  "Temporarily disable `hexl-mode' for OPERATION."
-  (when (derived-mode-p 'hexl-mode)
-    (hexl-mode-exit)
-    (set (make-local-variable 'vlf-restore-hexl-mode) operation)))
-
-(defun vlf-hexl-after (&optional operation)
-  "Re-enable `hexl-mode' if active before OPERATION."
-  (when (and (boundp 'vlf-restore-hexl-mode)
-             (eq vlf-restore-hexl-mode operation))
-    (hexl-mode)
-    (kill-local-variable 'vlf-restore-hexl-mode)))
-
-(add-hook 'vlf-before-batch-functions 'vlf-hexl-before)
-(add-hook 'vlf-after-batch-functions 'vlf-hexl-after)
-(add-hook 'vlf-before-chunk-update 'vlf-hexl-before)
-(add-hook 'vlf-after-chunk-update 'vlf-hexl-after)
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;; hexl mode integration
 
 (eval-after-load "hexl"
   '(progn
@@ -264,23 +256,34 @@ with the prefix argument DECREASE it is halved."
 
 (defun vlf-set-batch-size (size)
   "Set batch to SIZE bytes and update chunk."
-  (interactive (list (read-number "Size in bytes: " vlf-batch-size)))
+  (interactive
+   (list (read-number "Size in bytes: "
+                      (vlf-tune-optimal-load
+                       (if (derived-mode-p 'hexl-mode)
+                           '(:hexl :dehexlify :insert :encode)
+                         '(:insert :encode))))))
   (setq vlf-batch-size size)
   (vlf-move-to-batch vlf-start-pos))
 
 (defun vlf-beginning-of-file ()
   "Jump to beginning of file content."
   (interactive)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :dehexlify :insert :encode)
+                   '(:insert :encode)))
   (vlf-move-to-batch 0))
 
 (defun vlf-end-of-file ()
   "Jump to end of file content."
   (interactive)
   (vlf-verify-size)
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :dehexlify :insert :encode)
+                   '(:insert :encode)))
   (vlf-move-to-batch vlf-file-size))
 
-(defun vlf-revert (&optional _ignore-auto noconfirm)
-  "Revert current chunk.  Ignore _IGNORE-AUTO.
+(defun vlf-revert (&optional _auto noconfirm)
+  "Revert current chunk.  Ignore _AUTO.
 Ask for confirmation if NOCONFIRM is nil."
   (interactive)
   (when (or noconfirm
@@ -292,6 +295,9 @@ Ask for confirmation if NOCONFIRM is nil."
 (defun vlf-jump-to-chunk (n)
   "Go to to chunk N."
   (interactive "nGoto to chunk: ")
+  (vlf-tune-load (if (derived-mode-p 'hexl-mode)
+                     '(:hexl :dehexlify :insert :encode)
+                   '(:insert :encode)))
   (vlf-move-to-batch (* (1- n) vlf-batch-size)))
 
 (defun vlf-no-modifications ()