Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3215 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.0990. Min Range: -4.0990

Max Act: 4.0990. Min Act: -0.0000

Data Index: 1458352 (Python Code)

Max Activating Token Index: 758

Text #1

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.5837. Min Act: -0.0000

Data Index: 1604036 (Python Code)

Max Activating Token Index: 836

Text #2

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.8351. Min Act: -0.0000

Data Index: 1605875 (Python Code)

Max Activating Token Index: 392

Text #3

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.6476. Min Act: -0.0000

Data Index: 1386067 (Python Code)

Max Activating Token Index: 927

Text #4

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.5951. Min Act: -0.0000

Data Index: 1653191 (Python Code)

Max Activating Token Index: 999

Text #5

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.5307. Min Act: -0.0000

Data Index: 1563275 (Python Code)

Max Activating Token Index: 1005

Text #6

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.4970. Min Act: -0.0000

Data Index: 1363114 (Python Code)

Max Activating Token Index: 688

Text #7

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.4498. Min Act: -0.0000

Data Index: 1458676 (Python Code)

Max Activating Token Index: 131

Text #8

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.2318. Min Act: -0.0000

Data Index: 1489438 (Python Code)

Max Activating Token Index: 665

Text #9

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0672. Min Act: -0.0000

Data Index: 1637334 (Python Code)

Max Activating Token Index: 117

Text #10

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0487. Min Act: -0.0000

Data Index: 1563261 (Python Code)

Max Activating Token Index: 828

Text #11

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0255. Min Act: -0.0000

Data Index: 1653609 (Python Code)

Max Activating Token Index: 256

Text #12

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0177. Min Act: -0.0000

Data Index: 1505238 (Python Code)

Max Activating Token Index: 152

Text #13

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.1480. Min Act: -0.0000

Data Index: 1604595 (Python Code)

Max Activating Token Index: 181

Text #14

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.1071. Min Act: -0.0000

Data Index: 1582572 (Python Code)

Max Activating Token Index: 735

Text #15

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.1028. Min Act: -0.0000

Data Index: 1499094 (Python Code)

Max Activating Token Index: 507

Text #16

Max Range: 4.0990. Min Range: -4.0990

Max Act: 2.9418. Min Act: -0.0000

Data Index: 1611628 (Python Code)

Max Activating Token Index: 579

Text #17

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0668. Min Act: -0.0000

Data Index: 1567005 (Python Code)

Max Activating Token Index: 548

Text #18

Max Range: 4.0990. Min Range: -4.0990

Max Act: 2.9193. Min Act: -0.0000

Data Index: 1446349 (Python Code)

Max Activating Token Index: 280

Text #19

Max Range: 4.0990. Min Range: -4.0990

Max Act: 3.0379. Min Act: -0.0000

Data Index: 1517819 (Python Code)

Max Activating Token Index: 231