Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 421 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.9170. Min Act: -0.0000

Data Index: 637912 (The Pile)

Max Activating Token Index: 417

Text #1

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.3922. Min Act: -0.0000

Data Index: 877719 (The Pile)

Max Activating Token Index: 58

Text #2

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.3490. Min Act: -0.0000

Data Index: 973427 (The Pile)

Max Activating Token Index: 982

Text #3

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.1673. Min Act: -0.0000

Data Index: 720072 (The Pile)

Max Activating Token Index: 684

Text #4

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.3525. Min Act: -0.0000

Data Index: 1418613 (The Pile)

Max Activating Token Index: 1014

Text #5

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.2490. Min Act: -0.0000

Data Index: 90227 (The Pile)

Max Activating Token Index: 916

Text #6

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.4875. Min Act: -0.0000

Data Index: 1810789 (The Pile)

Max Activating Token Index: 90

Text #7

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.4593. Min Act: 0.0000

Data Index: 236279 (The Pile)

Max Activating Token Index: 686

Text #8

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.8825. Min Act: -0.0000

Data Index: 1747895 (The Pile)

Max Activating Token Index: 930

Text #9

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.2777. Min Act: -0.0000

Data Index: 460159 (The Pile)

Max Activating Token Index: 1007

Text #10

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9815. Min Act: -0.0000

Data Index: 461633 (The Pile)

Max Activating Token Index: 169

Text #11

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9311. Min Act: -0.0000

Data Index: 486135 (The Pile)

Max Activating Token Index: 1015

Text #12

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9261. Min Act: 0.0000

Data Index: 663526 (The Pile)

Max Activating Token Index: 58

Text #13

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.1706. Min Act: -0.0000

Data Index: 1562279 (The Pile)

Max Activating Token Index: 67

Text #14

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.1209. Min Act: -0.0000

Data Index: 1884528 (The Pile)

Max Activating Token Index: 1019

Text #15

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.0272. Min Act: -0.0000

Data Index: 1011250 (The Pile)

Max Activating Token Index: 842

Text #16

Max Range: 3.9170. Min Range: -3.9170

Max Act: 3.0945. Min Act: -0.0000

Data Index: 746686 (The Pile)

Max Activating Token Index: 227

Text #17

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9543. Min Act: -0.0000

Data Index: 1096012 (The Pile)

Max Activating Token Index: 935

Text #18

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9139. Min Act: 0.0000

Data Index: 273054 (The Pile)

Max Activating Token Index: 71

Text #19

Max Range: 3.9170. Min Range: -3.9170

Max Act: 2.9077. Min Act: -0.0000

Data Index: 845619 (The Pile)

Max Activating Token Index: 141