Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4833 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8361. Min Range: -3.8361

Max Act: 3.8361. Min Act: -0.0000

Data Index: 1884076 (The Pile)

Max Activating Token Index: 287

Text #1

Max Range: 3.8361. Min Range: -3.8361

Max Act: 3.2886. Min Act: -0.0000

Data Index: 1719107 (The Pile)

Max Activating Token Index: 390

Text #2

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.9314. Min Act: -0.0001

Data Index: 54393 (The Pile)

Max Activating Token Index: 362

Text #3

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.6861. Min Act: -0.0000

Data Index: 694610 (The Pile)

Max Activating Token Index: 176

Text #4

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.5421. Min Act: -0.0000

Data Index: 784396 (The Pile)

Max Activating Token Index: 659

Text #5

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.4072. Min Act: -0.0000

Data Index: 1720644 (The Pile)

Max Activating Token Index: 431

Text #6

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.3124. Min Act: -0.0000

Data Index: 1656004 (The Pile)

Max Activating Token Index: 371

Text #7

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.4267. Min Act: -0.0000

Data Index: 482920 (The Pile)

Max Activating Token Index: 795

Text #8

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.3983. Min Act: -0.0000

Data Index: 87963 (The Pile)

Max Activating Token Index: 825

Text #9

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.3975. Min Act: -0.0000

Data Index: 1115988 (The Pile)

Max Activating Token Index: 469

Text #10

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.1453. Min Act: -0.0000

Data Index: 1026526 (The Pile)

Max Activating Token Index: 201

Text #11

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.2272. Min Act: -0.0000

Data Index: 465163 (The Pile)

Max Activating Token Index: 487

Text #12

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.1373. Min Act: -0.0000

Data Index: 953163 (The Pile)

Max Activating Token Index: 300

Text #13

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.1399. Min Act: -0.0000

Data Index: 786301 (The Pile)

Max Activating Token Index: 410

Text #14

Max Range: 3.8361. Min Range: -3.8361

Max Act: 1.9971. Min Act: -0.0000

Data Index: 414167 (The Pile)

Max Activating Token Index: 641

Text #15

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.1056. Min Act: -0.0000

Data Index: 1663801 (The Pile)

Max Activating Token Index: 375

Text #16

Max Range: 3.8361. Min Range: -3.8361

Max Act: 1.9682. Min Act: -0.0000

Data Index: 913462 (The Pile)

Max Activating Token Index: 938

Text #17

Max Range: 3.8361. Min Range: -3.8361

Max Act: 2.0681. Min Act: -0.0000

Data Index: 895055 (The Pile)

Max Activating Token Index: 358

Text #18

Max Range: 3.8361. Min Range: -3.8361

Max Act: 1.9456. Min Act: -0.0000

Data Index: 65586 (The Pile)

Max Activating Token Index: 54

Text #19

Max Range: 3.8361. Min Range: -3.8361

Max Act: 1.8771. Min Act: -0.0000

Data Index: 1457985 (The Pile)

Max Activating Token Index: 863