Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3066 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.2271. Min Act: -0.0000

Data Index: 1796604 (The Pile)

Max Activating Token Index: 151

Text #1

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1934. Min Act: -0.0000

Data Index: 857527 (The Pile)

Max Activating Token Index: 400

Text #2

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1922. Min Act: -0.0000

Data Index: 488402 (The Pile)

Max Activating Token Index: 1014

Text #3

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1968. Min Act: -0.0000

Data Index: 294734 (The Pile)

Max Activating Token Index: 1003

Text #4

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1967. Min Act: -0.0000

Data Index: 901326 (The Pile)

Max Activating Token Index: 900

Text #5

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.0834. Min Act: -0.0000

Data Index: 1810546 (The Pile)

Max Activating Token Index: 332

Text #6

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1975. Min Act: -0.0000

Data Index: 762873 (The Pile)

Max Activating Token Index: 432

Text #7

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1346. Min Act: -0.0000

Data Index: 1623194 (The Pile)

Max Activating Token Index: 453

Text #8

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.0582. Min Act: -0.0000

Data Index: 1580825 (The Pile)

Max Activating Token Index: 857

Text #9

Max Range: 1.2271. Min Range: -1.2271

Max Act: 0.9436. Min Act: -0.0000

Data Index: 379340 (The Pile)

Max Activating Token Index: 874

Text #10

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1686. Min Act: -0.0000

Data Index: 454777 (The Pile)

Max Activating Token Index: 546

Text #11

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1360. Min Act: -0.0000

Data Index: 1013358 (The Pile)

Max Activating Token Index: 245

Text #12

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.2038. Min Act: -0.0000

Data Index: 1979004 (The Pile)

Max Activating Token Index: 599

Text #13

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.0901. Min Act: -0.0000

Data Index: 1803841 (The Pile)

Max Activating Token Index: 582

Text #14

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1186. Min Act: -0.0000

Data Index: 1196730 (The Pile)

Max Activating Token Index: 903

Text #15

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1528. Min Act: -0.0000

Data Index: 567883 (The Pile)

Max Activating Token Index: 918

Text #16

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1485. Min Act: -0.0000

Data Index: 147293 (The Pile)

Max Activating Token Index: 209

Text #17

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.0234. Min Act: -0.0000

Data Index: 1661200 (The Pile)

Max Activating Token Index: 154

Text #18

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.0120. Min Act: -0.0000

Data Index: 983063 (The Pile)

Max Activating Token Index: 242

Text #19

Max Range: 1.2271. Min Range: -1.2271

Max Act: 1.1020. Min Act: -0.0000

Data Index: 776867 (The Pile)

Max Activating Token Index: 469