Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 99 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1450. Min Act: -0.0002

Data Index: 1363884 (Python Code)

Max Activating Token Index: 578

Text #1

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1378. Min Act: -0.0002

Data Index: 1434394 (Python Code)

Max Activating Token Index: 688

Text #2

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1320. Min Act: -0.0002

Data Index: 1507080 (Python Code)

Max Activating Token Index: 642

Text #3

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1261. Min Act: -0.0002

Data Index: 1476999 (Python Code)

Max Activating Token Index: 975

Text #4

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1283. Min Act: -0.0002

Data Index: 1362444 (Python Code)

Max Activating Token Index: 866

Text #5

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1151. Min Act: -0.0002

Data Index: 1470463 (Python Code)

Max Activating Token Index: 139

Text #6

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1186. Min Act: -0.0002

Data Index: 1511845 (Python Code)

Max Activating Token Index: 269

Text #7

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1185. Min Act: -0.0002

Data Index: 1485846 (Python Code)

Max Activating Token Index: 539

Text #8

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1140. Min Act: -0.0002

Data Index: 1654936 (Python Code)

Max Activating Token Index: 587

Text #9

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1069. Min Act: -0.0002

Data Index: 1630802 (Python Code)

Max Activating Token Index: 258

Text #10

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1088. Min Act: -0.0002

Data Index: 1487079 (Python Code)

Max Activating Token Index: 875

Text #11

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1099. Min Act: -0.0002

Data Index: 1592968 (Python Code)

Max Activating Token Index: 804

Text #12

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1060. Min Act: -0.0002

Data Index: 1637453 (Python Code)

Max Activating Token Index: 203

Text #13

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1022. Min Act: -0.0002

Data Index: 1643683 (Python Code)

Max Activating Token Index: 864

Text #14

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1046. Min Act: -0.0002

Data Index: 1359899 (Python Code)

Max Activating Token Index: 820

Text #15

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1078. Min Act: -0.0002

Data Index: 1421363 (Python Code)

Max Activating Token Index: 738

Text #16

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1030. Min Act: -0.0002

Data Index: 1504499 (Python Code)

Max Activating Token Index: 200

Text #17

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1079. Min Act: -0.0002

Data Index: 1597626 (Python Code)

Max Activating Token Index: 469

Text #18

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1004. Min Act: -0.0002

Data Index: 1391096 (Python Code)

Max Activating Token Index: 569

Text #19

Max Range: 0.1450. Min Range: -0.1450

Max Act: 0.1044. Min Act: -0.0002

Data Index: 1487108 (Python Code)

Max Activating Token Index: 796