Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 78 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.9078. Min Act: -0.0000

Data Index: 1026145 (The Pile)

Max Activating Token Index: 279

Text #1

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.8101. Min Act: -0.0000

Data Index: 1431652 (The Pile)

Max Activating Token Index: 607

Text #2

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.8194. Min Act: -0.0000

Data Index: 1776594 (The Pile)

Max Activating Token Index: 506

Text #3

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7129. Min Act: -0.0000

Data Index: 710482 (The Pile)

Max Activating Token Index: 602

Text #4

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7818. Min Act: -0.0000

Data Index: 913100 (The Pile)

Max Activating Token Index: 637

Text #5

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7949. Min Act: -0.0000

Data Index: 1736158 (The Pile)

Max Activating Token Index: 113

Text #6

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7927. Min Act: -0.0000

Data Index: 905918 (The Pile)

Max Activating Token Index: 1004

Text #7

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.8132. Min Act: -0.0000

Data Index: 1548242 (The Pile)

Max Activating Token Index: 251

Text #8

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7860. Min Act: -0.0000

Data Index: 351264 (The Pile)

Max Activating Token Index: 780

Text #9

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7882. Min Act: -0.0000

Data Index: 1752874 (The Pile)

Max Activating Token Index: 137

Text #10

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7810. Min Act: -0.0000

Data Index: 323302 (The Pile)

Max Activating Token Index: 263

Text #11

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7366. Min Act: -0.0000

Data Index: 1432661 (The Pile)

Max Activating Token Index: 352

Text #12

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7695. Min Act: -0.0000

Data Index: 509712 (The Pile)

Max Activating Token Index: 65

Text #13

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7748. Min Act: -0.0000

Data Index: 1011006 (The Pile)

Max Activating Token Index: 582

Text #14

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7589. Min Act: -0.0000

Data Index: 1154900 (The Pile)

Max Activating Token Index: 23

Text #15

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7359. Min Act: -0.0000

Data Index: 643936 (The Pile)

Max Activating Token Index: 385

Text #16

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7560. Min Act: -0.0000

Data Index: 1066298 (The Pile)

Max Activating Token Index: 463

Text #17

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7776. Min Act: -0.0000

Data Index: 673068 (The Pile)

Max Activating Token Index: 952

Text #18

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7322. Min Act: -0.0000

Data Index: 1441768 (The Pile)

Max Activating Token Index: 677

Text #19

Max Range: 0.9078. Min Range: -0.9078

Max Act: 0.7261. Min Act: -0.0000

Data Index: 1005195 (The Pile)

Max Activating Token Index: 47