Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2187 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.3384. Min Range: -3.3384

Max Act: 3.3384. Min Act: -0.0000

Data Index: 712985 (The Pile)

Max Activating Token Index: 561

Text #1

Max Range: 3.3384. Min Range: -3.3384

Max Act: 3.0135. Min Act: -0.0000

Data Index: 915951 (The Pile)

Max Activating Token Index: 148

Text #2

Max Range: 3.3384. Min Range: -3.3384

Max Act: 2.1623. Min Act: -0.0000

Data Index: 389720 (The Pile)

Max Activating Token Index: 464

Text #3

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.9775. Min Act: -0.0000

Data Index: 824211 (The Pile)

Max Activating Token Index: 448

Text #4

Max Range: 3.3384. Min Range: -3.3384

Max Act: 2.0338. Min Act: -0.0000

Data Index: 285887 (The Pile)

Max Activating Token Index: 272

Text #5

Max Range: 3.3384. Min Range: -3.3384

Max Act: 2.0530. Min Act: -0.0000

Data Index: 258023 (The Pile)

Max Activating Token Index: 248

Text #6

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8450. Min Act: -0.0000

Data Index: 1940982 (The Pile)

Max Activating Token Index: 987

Text #7

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8171. Min Act: -0.0000

Data Index: 75575 (The Pile)

Max Activating Token Index: 308

Text #8

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.9794. Min Act: -0.0000

Data Index: 1961858 (The Pile)

Max Activating Token Index: 957

Text #9

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8037. Min Act: -0.0000

Data Index: 188060 (The Pile)

Max Activating Token Index: 122

Text #10

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8964. Min Act: -0.0000

Data Index: 1620180 (The Pile)

Max Activating Token Index: 447

Text #11

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.6740. Min Act: -0.0000

Data Index: 1211108 (The Pile)

Max Activating Token Index: 441

Text #12

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.9466. Min Act: -0.0000

Data Index: 719555 (The Pile)

Max Activating Token Index: 806

Text #13

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8246. Min Act: -0.0000

Data Index: 1335771 (The Pile)

Max Activating Token Index: 688

Text #14

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.7137. Min Act: -0.0000

Data Index: 946737 (The Pile)

Max Activating Token Index: 472

Text #15

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.8545. Min Act: -0.0000

Data Index: 52293 (The Pile)

Max Activating Token Index: 654

Text #16

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.6226. Min Act: -0.0000

Data Index: 536987 (The Pile)

Max Activating Token Index: 698

Text #17

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.6117. Min Act: -0.0000

Data Index: 924538 (The Pile)

Max Activating Token Index: 448

Text #18

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.5185. Min Act: -0.0000

Data Index: 1748218 (The Pile)

Max Activating Token Index: 275

Text #19

Max Range: 3.3384. Min Range: -3.3384

Max Act: 1.6430. Min Act: -0.0000

Data Index: 1043142 (The Pile)

Max Activating Token Index: 694