Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4149 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.7680. Min Act: -0.0000

Data Index: 584467 (The Pile)

Max Activating Token Index: 580

Text #1

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.6240. Min Act: -0.0000

Data Index: 1518163 (The Pile)

Max Activating Token Index: 131

Text #2

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.1035. Min Act: -0.0000

Data Index: 1236099 (The Pile)

Max Activating Token Index: 987

Text #3

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.2536. Min Act: -0.0000

Data Index: 1367983 (The Pile)

Max Activating Token Index: 196

Text #4

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.4946. Min Act: -0.0000

Data Index: 417559 (The Pile)

Max Activating Token Index: 243

Text #5

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.3460. Min Act: -0.0000

Data Index: 691276 (The Pile)

Max Activating Token Index: 497

Text #6

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.0671. Min Act: -0.0000

Data Index: 383207 (The Pile)

Max Activating Token Index: 1005

Text #7

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.1605. Min Act: -0.0000

Data Index: 1194619 (The Pile)

Max Activating Token Index: 778

Text #8

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.1952. Min Act: -0.0000

Data Index: 795245 (The Pile)

Max Activating Token Index: 159

Text #9

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.8633. Min Act: -0.0000

Data Index: 1225439 (The Pile)

Max Activating Token Index: 179

Text #10

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.0076. Min Act: -0.0000

Data Index: 198359 (The Pile)

Max Activating Token Index: 118

Text #11

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.7986. Min Act: -0.0000

Data Index: 978223 (The Pile)

Max Activating Token Index: 371

Text #12

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.0633. Min Act: -0.0000

Data Index: 1395292 (The Pile)

Max Activating Token Index: 1010

Text #13

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.1998. Min Act: -0.0000

Data Index: 1052708 (The Pile)

Max Activating Token Index: 231

Text #14

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.9507. Min Act: -0.0000

Data Index: 737190 (The Pile)

Max Activating Token Index: 653

Text #15

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.9295. Min Act: -0.0000

Data Index: 1334793 (The Pile)

Max Activating Token Index: 228

Text #16

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.8025. Min Act: -0.0000

Data Index: 1487462 (The Pile)

Max Activating Token Index: 987

Text #17

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.8991. Min Act: -0.0000

Data Index: 1144962 (The Pile)

Max Activating Token Index: 542

Text #18

Max Range: 3.7680. Min Range: -3.7680

Max Act: 3.0810. Min Act: -0.0000

Data Index: 1439483 (The Pile)

Max Activating Token Index: 998

Text #19

Max Range: 3.7680. Min Range: -3.7680

Max Act: 2.7605. Min Act: -0.0000

Data Index: 1296824 (The Pile)

Max Activating Token Index: 213